[gentoo-dev] Portage QOS

[gentoo-dev] Portage QOS

[LTHR]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]

Hi All,

What do you think about implementing this:

http://forums.gentoo.org/viewtopic.php?p=7477494

I've system design in my head and could write it down with the implementation details.
Then may be we could all review it and get to something we all agree upon then I could 
try getting a team and implement it.

Just a brief question - does anyone know how many ebuilds are assembled world 
wide each second?

-- 
Best regards,
 LTHR                          mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 1199 bytes --]

Re: [gentoo-dev] Portage QOS

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 4998 bytes --]

On Wed, Jan 8, 2014 at 11:24 PM, LTHR <lanthruster@gmail.com> wrote:

>  Hi All,
>
>
I want to start off by discussing your premise, before embarking on the
overall goals.

You wrote:
"I'm with Gentoo for many years. For various reasons many techs were not
implemented and now Gentoo is in a kind of stagnation. But we can give
Gentoo a new birth with relatively little efforts and bring the distro to
the whole new level. "

I don't actually believe your premise of stagnation. But I can put aside my
disagreement for now. Lets talk about how the overall goal of 'bringing the
distro to a whole new level' and how 'Portage QoS' will help us get there.
I don't think you covered these points well in your post (I will talk about
the goals more below...)


> What do you think about implementing this:
>
> http://forums.gentoo.org/viewtopic.php?p=7477494
>
I've system design in my head and could write it down with the
> implementation details.
> Then may be we could all review it and get to something we all agree upon
> then I could
> try getting a team and implement it.
>

Later in your post you wrote about the goals for Porage QoS.

Time we spare time for everyone - users, developers, maintainers.
Quality in 3-5 years we improve GENTOO in a way it will be in top10
distros, the users will be happy
Automatization no time to waste to improve Gentoo for the community,
everyone with GENTOO is part of GENTOO working for GENTOO
Bug Tracking no more we spare resources deployed in the bug tracking
system. It will exist. But's it's will be extended with robotic help from
Portage QOS
Knowledge we will know exactly who, how in what way GENTOO is used and we
will create a system for USERS not vice-versa
Order we will know exactly where to go next and what to do next what focus
on next
Integrity all GENTOO users will be able to participate in project. No
matter what experience they have. We will utilize help of a great number of
supporters.

Time: Portage QoS will save everyone time. I can actually believe this in
an ideal world where developers built automation around a system like
Portage QoS. Ironically I think tool development for *developers* is an
area that we are terrible at. Perhaps Portage QoS will have an awesome
easy-to-use API that makes tool writing a breeze, I don't really know. I
don't think blaming the portage API is necessarily the key to 'all our
tools are terrible' though.

Quality: Portage QoS will improve 'quality'. Again though, we don't really
measure quality in any quantitative way. If we switch to automated
reporting of failures, quality will actually go down, if we count quality
as 'reports of problems' because now reports are automated, rather than
manual. I think the big fear here is that many teams are already
understaffed and the automated system will quickly drown them. I imagine
Portage QoS could solve the 'drowning' problem, but i haven't see many
systems handle it well.

Bug tracking: Spend less resources on bug tracking. I think there is a lot
of missed opportunity for bugs automation. The sad fact is that infra is
terrible in areas like this, because the bug system is very opaque for
non-infra folks and the infra folks involved are not interested / don't
have time to implement the automation. So I will nominally agree here (even
if the automation isn't necessarily Portage QoS;e.g. we have discussed
automated bug assignment for *years*)

Knowledge: So I think in general I agree, insofar as more information can
be helpful in making decisions. I think you should take note that there are
at least 4-5 'gentoo stats' projects that have been tried and my
understanding is that none of them are in operation today.

Future Planning (you wrote: Order): I think this segment sort of
illustrates a misunderstanding of the Gentoo project as it is today. I
don't think developers are necessarily 'confused at how Gentoo is used' or
'do not know what to work on next.' I think developers work on whatever
they find interesting (that is what I do anyway ;p)

Back to the premise of bringing the distro to a whole new level. Some of
the items above I think have merits on their own, but they still don't
guide me to your ultimate goal. You outlined some of what I presume to be
defects in Gentoo today.

Package blocks that portage does not resolve automatically
Slot conflicts that portage does not resolve automatically
Compile failures
...What else?

So part of the Portage QoS system is that users will submit their failures
and Portage QoS will serve as a knowledge base of known issues. To me, that
is still a pretty bad User Experience. Can't we just get portage to handle
these issues transparently, as per
http://forums.gentoo.org/viewtopic-t-977936.html

Just a brief question - does anyone know how many ebuilds are assembled
> world
> wide each second?
>

I bet you more apks are installed per second ;)

-A


>
>
> *--  Best regards,  LTHR                          *
> mailto:lanthruster@gmail.com <lanthruster@gmail.com>
>

[-- Attachment #2: Type: text/html, Size: 7598 bytes --]

Re: [gentoo-dev] Portage QOS

[Igor]

[-- Attachment #1: Type: text/plain, Size: 8116 bytes --]

Hello Alec,

Thursday, January 9, 2014, 12:12:18 PM, you wrote:


On Wed, Jan 8, 2014 at 11:24 PM, LTHR <lanthruster@gmail.com> wrote:
Hi All,


I want to start off by discussing your premise, before embarking on the overall goals.

You wrote:
"I'm with Gentoo for many years. For various reasons many techs were not implemented and now Gentoo is in a kind of stagnation. But we can give Gentoo a new birth with relatively little efforts and bring the distro to the whole new level. "

I don't actually believe your premise of stagnation. But I can put aside my disagreement for now.


True. There is no data to tell what happens with Gentoo (to give that data is one of the goals of the 
project). We only have some formal esteems from unreliable sources. 

According to distro watch: 

http://web.archive.org/web/20130701000000*/http://distrowatch.com/dwres.php?resource=popularity

In February 2012, Gentoo distro was in 19th place. 
In December 2012, Gentoo went to 22nd place. 
In December 2013, Gentoo is down to 32nd place 

According to Linux Counter 
http://web.archive.org/web/20120101000000*/http://linuxcounter.net/distributions/stats.html

In January 2012, Gentoo distro had 5.32%
In January 2012, Gentoo had 4.04%
In November 2013, Gentoo had 4,21% 

And from my experience of Gentoo forums, gentoo.wiki - I vote for Gentoo at least not gaining new users.
If in several years the number of users is not increased - we can tell about stagnation. 

Why new users are not with Gentoo and how to get them - good question, let's find out!


 Lets talk about how the overall goal of 'bringing the distro to a whole new level' and how 'Portage QoS' will help us get there. I don't think you covered these points well in your post (I will talk about the goals more below...)


I'll re-phrase the goals. 

It's all not very well thought after at this stage but immediate goals are like this: 


* Knowledge of the number of Gentoo distros installed world wide - knowing the trend how many users choose 
  Gentoo and where Gentoo is really going down|up|stands still. 
  
  You can then try different features and see how a feature is met - if the number of systems increase 
  then this feature is probably useful. It's a strategical job, somebody at the very top of the project should 
  analyze databases and make conclusions. 

* Knowledge of the ebuild popularity - what ebuilds are popular and what are not - what ebuild to give an extra focus 
  and what ebuild could wait

* Knowledge of ebuild quality. If some ebuilds fail on many systems - something is wrong and ebuild and may be portage 
  need fixing. It's especially useful to make sure that all ebuilds have correct dependencies, missing dependencies, etc.

* A formal esteem of portage quality 
  PortageQ = (the number of successful ebuilds/the number of all ebuild attempts)

  Portage speed efficiency:  
  Average time before build starts (or download starts)

  How many times portage fails itself. 

* Immediate problem detection. If the number of PortageQ went down last day - there is some problem. 
  (then you go to ebuild stats and see what is failing)

* Reducing load on bugtracker folks - the build problems will be detected automatically and solved according 
  to their importance. There will be no need to supply bug tracker with ebuild logs and emerge --info if 
  somebody wants to report a problem. 

* Team efficiency esteem. The stats will tell what ebuilds are failing most often. 

* Team automated info. When failure rate of a certain ebuild increase the maintainer is automatically 
  informed and he can login in web-interface and see details how exactly ebuild failed. 
  The same for the portage itself. Next day a maintainer could push a new ebuild in the portage and the 
  problem might be solved.

  It's not possible not to make mistakes. But it's possible to react on their consequences fast. 

* Knowledge what kernels are used by Gentoo users, how often they update their systems, what flags 
  are used 

2nd turn goals: 

* to integrate forums.gentoo.org and bug tracker. People are offering great workarounds and solutions. But 
  they're not known to the majority of Gentoo users. 

  If a e-build fails - may be there is already a solution - and we can offer the solutions automatically from 
  portage. Like:

  There might be some work-arounds on this problem: 
  [Gentoo Forum - qt-core ebuild fails - SOLVED] 
  htpp://forums.gentoo.org/..... 

  There is a known bug on this ebuild: 
  [Gentoo Bug - qt-core ebuild fails] 
  htpp://forums.gentoo.org/..... 

* to make Bug Tracker almost unmanned. We can use gathered infromation on failed e-builds to 
  create bugs in Bug Tracker automatically and automatically set priorities according to the 
  severity. 

  The severity could be assigned automatically from package popularity and failure rate stats.

  The users with the problems could receive e-mail automatically to follow up the quick arounds
  and solutions. 

  No need to change bug-tracker version, you just need a robot who would maintain bug-tracker 
  analyazing PortageQOS databases. 


I'm sure there could be more applications of the system. 



Back to the premise of bringing the distro to a whole new level. Some of the items above I think have merits on their own, but they still don't guide me to your ultimate goal. You outlined some of what I presume to be defects in Gentoo today.


It's hard to see more goals at this stage. It's like when you're building a hammer to set a nail and 
then when you have it you discover that you may use it to pull a nail too. It's for the future developers 
to decide. 

Knowledge: So I think in general I agree, insofar as more information can be helpful in making decisions. I think you should take note that there are at least 4-5 'gentoo stats' projects that have been tried and my understanding is that none of them are in operation today.

True, one of the reasons - the whole system planning shouldn't be apart from Gentoo developers. 

I could write the whole system on paper, with tools, soft used and the system design not narrowing to the table structures.
Then everyone could contribute his experience to it or see a problem at design time. 

I know how to design the system to make handling significant load. It has a good chance 
to work fast and to be scalable. 


With the design approved by others, programming is easy. 

It's not going to be very complex.
I estimate the whole system as 10 000 - 15 000 lines max in different languages. The most difficult part would 
be to make it scalable and fast as we don't know how many ebuilds are done world wide per second and we 
have to be prepared to hot-plug hardware at any time not redesigning the system. And the system shouldn't get 
overloaded/loose functionality. 



Package blocks that portage does not resolve automatically
Slot conflicts that portage does not resolve automatically
Compile failures
...What else?

So part of the Portage QoS system is that users will submit their failures and Portage QoS will serve as a knowledge base of known issues. To me, that is still a pretty bad User Experience. Can't we just get portage to handle these issues transparently, as per http://forums.gentoo.org/viewtopic-t-977936.html




If we ask users about questions they don't know answers to it will result in users loss. But it's totally 
out of my expertise if the portage interaction is GOOD 

Users are not doing anything usual than they do right now. The reports are sent transparently by 
portage to the PorageQOS load balancer and the suggestions are received.



Just a brief question - does anyone know how many ebuilds are assembled world 
wide each second?

I bet you more apks are installed per second ;)

-A
 
-- 
Best regards,
 LTHR                          mailto:lanthruster@gmail.com




-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 11794 bytes --]

Re: [gentoo-dev] Portage QOS

[Christopher Schwan]

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

Hi,

you motivate your proposal by claiming the Gentoo Project stagnates which you 
relate with its decline in popularity:

> According to Linux Counter
> http://web.archive.org/web/20120101000000*/http://linuxcounter.net/distribut
> ions/stats.html
> 
> In January 2012, Gentoo distro had 5.32%
> In January 2012, Gentoo had 4.04%
> In November 2013, Gentoo had 4,21%
> 
> And from my experience of Gentoo forums, gentoo.wiki - I vote for Gentoo at
> least not gaining new users. If in several years the number of users is not
> increased - we can tell about stagnation.

But let me ask this question: Is the number of users really that important to 
Gentoo? Since it does not strive for world domination I think all that matters 
is to keep the current userbase happy. From your thread I do not understand 
whats wrong on that side:

> For various reasons many techs were not implemented and now Gentoo is in a 
kind of stagnation.

What do you mean by that in particular? And what is wrong with 
bugs.gentoo.org? Wouldn't it be better to talk about how attract more 
developers? I guess a lot unsolved bugs stem from the fact that there are too 
few that can take care of them.

Cheers,
Christopher

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Portage QOS

[Igor]

[-- Attachment #1: Type: text/plain, Size: 4357 bytes --]

Hello Christopher,

Thursday, January 9, 2014, 6:12:37 PM, you wrote:


> you motivate your proposal by claiming the Gentoo Project stagnates which you
> relate with its decline in popularity:

>> According to Linux Counter
>> http://web.archive.org/web/20120101000000*/http://linuxcounter.net/distribut
>> ions/stats.html

>> In January 2012, Gentoo distro had 5.32%
>> In January 2012, Gentoo had 4.04%
>> In November 2013, Gentoo had 4,21%

>> And from my experience of Gentoo forums, gentoo.wiki - I vote for Gentoo at
>> least not gaining new users. If in several years the number of users is not
>> increased - we can tell about stagnation.

> But let me ask this question: Is the number of users really that important to
> Gentoo? 

Should be. It looks important for Gentoo and the Gentoo developers.

For Gentoo - the more users Gentoo has the more resources it can deploy 
in development and support. May be the world domination isn't the correct 
word but a steady growth of Gentoo systems globally is a good goal. If Gentoo 
looses 1% annually in 5 years there would be no new developers motivated enough 
to push the project ahead and the old ones would think bad about investing their 
life in what is not gaining popularity and they will not be that enthusiastic, 
getting demoralized. Without fresh blood the crucial people will get old/tired and alas. 
I'm sure you don't need Gentoo only for yourself as nobody else here does. It's a project 
for people, the people get it running and it's the treasure the Gentoo has so why not 
to turn towards users? 

Try build a house, tell friends about the house, unite them in one goal and 
then build it for free and for everybody, then if the house is not working well 
- you have friends no more, next time you're alone in one huge house, 
nobody needs, nobody will help and you can't just support it on your own. 

> Since it does not strive for world domination I think all that matters
> is to keep the current userbase happy.

True. That is the goal.
 
> From your thread I do not understand
> whats wrong on that side:

>> For various reasons many techs were not implemented and now Gentoo is in a 
> kind of stagnation.

> What do you mean by that in particular? 

Gentoo stopped. The work is done but it's not a game changer. The 
ebuilds have approximately the same time to install, the failure rate is about the same, 
emerge is getting slower. The number of users decrease. 
It looks like this is because it's not clear where to go and what to change. What to change 
exactly to bring more users? Not clear. No information.

Apparently the criteria is timing. When you develop a human access interface the most 
reliable thing to check your work against is the time required for an average user to achieve the goal. 

Time is the most important in our lifes and this is the criteria that always works. There 
are a variety of users, with different genetics, different views, education and skills but you 
can find an interface that unites them all and everyone is feeling like it's easy. It's not an
easy task because what looks easy for a developer might look alien for his neighbour. 

If we decrease time required for the users to maintain Gentoo and decrease time for developers 
to push the project - then Gentoo will grow.

> And what is wrong with 
> bugs.gentoo.org? Wouldn't it be better to talk about how attract more 
> developers?  

Good question. 
You can't attract enough supporters not being successful or
without paying them. Supporters are the same users if you're loosing
users the number of supporters are decreasing.

The times are changed. The projects are so complicated nowadays that 
keeping them manually is practically impossible. Why drudgery? There 
are numerous jobs with which robots can do better. Human should focus 
on what machines can't do better. 

> I guess a lot unsolved bugs stem from the fact that there are too
> few that can take care of them.

And from all these bugs only 10% are critical 20% are affecting like 
1% of all users and it's not clear what to fix first. 

It's a self balancing system. How do you plan to attract more developers? 
What to offer them?

-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 6315 bytes --]

Re: [gentoo-dev] Portage QOS

[Ben Kohler]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

On Thu, Jan 9, 2014 at 6:44 AM, Igor <lanthruster@gmail.com> wrote:

>
>
> According to distro watch:
>
> ...
> According to Linux Counter
>
> ...
>

"What are distro watch and linux counter and who cares what their opt-in
stats gathering says?"
-most Gentoo users I've ever talked to

I think if you drop the premise "Gentoo is dying, how do we fix it?" and
just focus on "Here are some ideas on how we can improve Gentoo", you'll
get a better response here.  From what I see (IRC activity, new ebuild
activity, bugzilla activity, etc), our community seems stronger than ever.
 I think a lot of us are puzzled that you think Gentoo has "stopped".

You have some great ideas but this is not a sinking ship scenario.

-Ben

[-- Attachment #2: Type: text/html, Size: 1357 bytes --]

Re: [gentoo-dev] Portage QOS

[Jeroen Roovers]

On Thu, 9 Jan 2014 19:26:24 +0400
Igor <lanthruster@gmail.com> wrote:

> >> For various reasons many techs were not implemented and now Gentoo
> >> is in a 
> > kind of stagnation.
> 
> > What do you mean by that in particular? 
> 
> Gentoo stopped.

https://bugs.gentoo.org/show_bug.cgi?id=298754
https://bugs.gentoo.org/show_bug.cgi?id=439378
https://bugs.gentoo.org/show_bug.cgi?id=455908
https://bugs.gentoo.org/show_bug.cgi?id=495002

It looks like the only thing that is stagnating is your Gentoo Linux
install.


     jer

Re: [gentoo-dev] Portage QOS

[Igor]

Hello Ben,

Thursday, January 9, 2014, 7:49:28 PM, you wrote:

True, thanks for noting that.

> "What are distro watch and linux counter and who cares what their opt-in stats gathering says?"

> -most Gentoo users I've ever talked to

> I think if you drop the premise "Gentoo is dying, how do we fix
> it?" and just focus on "Here are some ideas on how we can improve
> Gentoo", you'll get a better response here.  From what I see (IRC
> activity, new ebuild activity, bugzilla activity, etc), our
> community seems stronger than ever.  I think a lot of us are puzzled
> that you think Gentoo has "stopped".  
>  


> You have some great ideas but this is not a sinking ship scenario.



-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

Re: [gentoo-dev] Portage QOS

[Igor]

Hello Jeroen,

Thursday, January 9, 2014, 7:55:42 PM, you wrote:

I was expecting you a few hours earlier, Jeroen. I knew you
wouldn't resist a terrible temptation remembering the Python Bug
that I filed from the old kernel gentoo.

For your information this is a confirmed bug in Python right now and
it is going to be fixed in 3.5

http://bugs.python.org/issue20065

Jeroen, tell me how many users world wide do not prefer to upgrade Gentoo
on automated basis? There are important servers, and there are many
cases when after upgrade server stops. Do you remember that recent udev
change? And there are many similar cases. Imagine that your server
is running a reactor. So what would you prefer to keep it running the
reactor as it did flawlessly for 8 years or launch an upgrade taking
the risks to blast yourself?

Many be it's not only me, but somebody else who is thinking the same?
Are you sure that the majority of Gentoo users are indulged in
paranoid automated upgrade and then spending time fixing damage
that upgrade did?

Do you have a car? Why you don't change EVERY detail in your car on a new
version on daily basis automatically?

Why don't you change car as soon as a new version is released? Why not
changing the new mouse, new keyboard, new monitor, new supply daily as
soon as there is a new version?

Not to mention that you can change daily appearances.

I belive that each users has the right to decide how to use Gentoo.
I have my reasons not to upgrade some servers.

And you, being a support team has no right to mock on your users
weather they're right or not. You don't make users feel bad, you
help them. One mocked user, another and there is no users to mock
about.

And you need users to mock more than they need you.

>> >> For various reasons many techs were not implemented and now Gentoo
>> >> is in a 
>> > kind of stagnation.
>> 
>> > What do you mean by that in particular? 
>> 
>> Gentoo stopped.

> https://bugs.gentoo.org/show_bug.cgi?id=298754
> https://bugs.gentoo.org/show_bug.cgi?id=439378
> https://bugs.gentoo.org/show_bug.cgi?id=455908
> https://bugs.gentoo.org/show_bug.cgi?id=495002

> It looks like the only thing that is stagnating is your Gentoo Linux
> install.





-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[gentoo-dev] Re: Portage QOS

[Duncan]

Igor posted on Thu, 09 Jan 2014 16:44:02 +0400 as excerpted:

> There is no data to tell what happens with Gentoo (to give that data is
> one of the goals of the project). We only have some formal esteems from
> unreliable sources.
> 
> According to distro watch:
> 
> In February 2012, Gentoo distro was in 19th place.
> In December 2012, Gentoo went to 22nd place.
> In December 2013, Gentoo is down to 32nd place

There was some discussion of this previously.  The conclusion was 
basically that gentooers don't tend to be the trend-watching type, nor, 
really, are we really interested in the trend-watching type, as that's 
not gentoo's base or target user. Instead, our users tend to be 
independent customizers that aren't so interested in what the majority 
thinks, but, rather find gentoo's general support for letting them make 
of their gentoo installation what they will a very good match for their 
needs.  If that's not the best match or if their needs change, the fact 
that gentoo takes more work than many distros because you have to 
actually configure and build it, tends to have them quickly off to some 
other distro that's a better fit for their less time/interest, more 
cookie-cutter needs.

In a way, then, gentoo in the Linux ecosystem is what Linux itself is in 
the more general OS ecosystem, and gentoo tends to get only the self-
selecting independent thinkers who value the ability to make their OS 
what they want while never-the-less automating much of the process (thus 
we aren't Linux from Scratch), in the same way that the same group, but 
to a somewhat lessor extent, tend to be Linux users.

And just as a significant subset of those Linux users and devs value 
their (software) freedom and independence enough to not be willing to 
sacrifice it just to have Linux more popular and maybe exceed MS, so a 
lot of Gentoo users and devs aren't willing to compromise on gentoo's 
ideals of highly customizable individuality just to see us rise in 
rankings such as distro-watch.  If it happens, great, but it won't 
greatly affect the way gentoo is developed, and if it doesn't happen, no 
big deal anyway, since that's not something we consider significant or 
important, particularly /because/ we recognize that sort of user isn't 
what gentoo's targeting in the first place.

> According to Linux Counter
> 
> In January 2012, Gentoo distro had 5.32%
> In January 2012, Gentoo had 4.04%
> In November 2013, Gentoo had 4,21%

I guess one of those January 2012s is supposed to be something 
different...

Same thing here, really, tho the reason is a bit different.

I know *I* certainly haven't registered with linuxcounter, and don't 
expect I ever will, either.  I see it as useless at best, and yet another 
way to be tracked at worst.  (I /do/ deliberately keep my browser's user-
agent string set to Linux instead of setting it to say the latest MS 
Windows version, and deliberately kept 64-bit back when 32-bit was the 
norm for similar reasons, so sites that I visit and thus care about can 
count that, but I most certainly do NOT let the various third-party 
tracing sites do their thing, using tools such as firefox plugins 
noscript, request-policy and cookie permissions, as well as privoxy, to 
help me keep that information out of third-party-tracker's hands.)

Tho interestingly, that does show percentage stabilizing or even 
increasing a bit between the second and third samples.  What it means, 
however, I'm not going to attempt to guess.  For all I know it simply 
means a few gentooers don't object to being tracked as strongly as they 
once did, which is actually slightly disturbing to me, tho it's their 
life so they get to decide, not me.

> And from my experience of Gentoo forums, gentoo.wiki - I vote for Gentoo
> at least not gaining new users.

That would be a more interesting number, there.  But you don't provide 
stats for that one, and personal perception such as yours above for those 
constantly involved is notoriously inaccurate.  Someone who left for a 
couple years and came back tends to see changes much better, for the same 
reason you don't tend to notice changes in a friend as you grow old 
together, unless you're separated for a few years and then meet again.

I wonder what the forums stats counts are.  I know there's mailing list 
activity stats as I've seen them posted occasionally, but I'm not sure if 
there's anything like that for the forums...  That would give us some 
concrete numbers to work with.

> If in several years the number of users is not increased - we can tell
> about stagnation.

As I've personally argued about Linux, if popularity comes at the cost of 
loss of freedom, etc, it's not worth it.  There's worse things than 
seeing numbers stagnate, and losing our ideals in a likely futile pursuit 
of popularity (what's the chances of gentoo topping Red Hat even if we 
forsook all that makes gentoo gentoo and tried? that's not what we're 
good at or care about) is one of them.

> It's all not very well thought after at this stage but immediate goals
> are like this:
> 
> 
> * Knowledge of [... multiple suggestions for tracking various things 
potentially objectionable to gentoo users.]

I suspect that the various gentoo stats efforts failed for the same 
reason I suspect fewer than normal gentoo users are registered with 
linuxcounter... Gentoo users tend to be the independent sort, and have a 
distinct aversion to being counted or tracked.  A few might opt in, but 
not enough to get particularly good or reliable data, and if it were opt-
out or worse-yet hard-coded, we'd likely lose a lot of gentoo users over 
it, not just because of the tracking objection itself (many can patch 
that out if they have too, as I did gentoo/kde's hard-coded semantic-
desktop stuff here, when they tried to dump the flag in early kde 4.11, 
tho fortunately they returned it before 4.11 stabilized), but because 
that sort of hard-coding would be a betrayal of everything that a lot of 
gentoo users have come to gentoo FOR, so were it to happen, it'd be time 
to leave.


Meanwhile, those of us who have been around gentoo for a few years have 
seen the "gentoo's stagnating/dying and here's what it must do to be-
popular-again/survive" thread several times over, by now.   Gentoo's 
still here; I'm still here.  Those ideas... aren't... until they come 
around for another round, as they seem to do every couple years...

And FWIW, gentoo's number of devs rose until it hit something around 300, 
then it fell back a bit (I think it reached 350 or so before they started 
actively retiring devs who had disappeared for quite some time, but I 
believe the number of active devs has never much exceeded 300, if that), 
but has remained relatively steady around 250-ish devs, 200 or so active 
depending on definition of "active", for several years now.  Sometimes it 
goes down a few, then it goes up a few, but overall it remains about the 
same.

Which actually fits various organizational/group dynamics models, 
apparently, too.  There's (apparently, this was posted one of the other 
times a discussion like this came up, and it makes sense, but I've not 
looked into it further) some studies to the effect that there are several 
group size thresholds.  IIRC (and I may not) the maximum effective size 
for small groups was 20-50.  At about that point, conflict goes up and 
groups either adapt and change their practices to grow, or drop down 
below that number again and tend to stay there.  There's another such 
threshold at 250-350, forcing further group practices adaption to grow 
further.  A number of FLOSS groups reach that one and never pass it, and 
gentoo has been right there for some years now.  But if that threshold is 
passed, the group can grow relatively unrestrained again, to a size of 
several thousand.  I believe Debian is one of the few all-community 
examples of passing the 250-350 threshold, but that they're stuck at the 
next one, 2500-3000.

I think the kernel has passed the 250-350 barrier now too, with git and 
the distributed hierarchy of kernel lieutenants, codified signed-off-by 
practices, etc, helping with that.  Before the distributed git and 
bitkeeper before that on the technical side, however, and before the 
hierarchy of kernel lieutenants was established, there was a real crisis, 
as Linus really was becoming the bottleneck, and nobody really knew how 
to fix it as there's only so much one man can do.  But they worked thru 
the issues and busted that cap, and the kernel's moving faster than ever 
thought possible, before.

If that is indeed the case, then in ordered to grow, gentoo needs to 
figure out how to get past that 250-350 developer threshold, and until/
unless we do, we'll "stagnate" in at least active developer numbers.  IOW, 
it's primarily a social/organizational problem, not a technical problem, 
tho as with the kernel and bitkeeper and then git, the right technical 
tools can help.

Actually, in that regard it's very possible that gentoo's long planned 
and worked toward cvs-to-git conversion will help finally bust that 
barrier for gentoo as well.  Time will tell I guess, but that's one more 
reason to try to help make it happen.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Portage QOS

[yac]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

On Thu, 9 Jan 2014 11:24:25 +0400
LTHR <lanthruster@gmail.com> wrote:

> Hi All,
> 
> What do you think about implementing this:
> 
> http://forums.gentoo.org/viewtopic.php?p=7477494
> 
> I've system design in my head and could write it down with the
> implementation details. Then may be we could all review it and get to
> something we all agree upon then I could try getting a team and
> implement it.
> 
> Just a brief question - does anyone know how many ebuilds are
> assembled world wide each second?
> 

Hi,

There are some ideas that may be worth pursuing and by bottom up
approach we (me and mainly naota) started turning gentwoo [1]_ [2]_ into
a package usage stats [3]_

.. [1] http://gentwoo.elisp.net/
.. [2] https://github.com/naota/gentwoo
.. [3] https://github.com/gentoo/GenTwoo-backend

---
Jan Matějka        | Gentoo Developer
https://gentoo.org | Gentoo Linux
GPG: A33E F5BC A9F6 DAFD 2021  6FB6 3EBF D45B EEB6 CA8B

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Portage QOS

[Igor]

Hello Duncan,

Thursday, January 9, 2014, 9:59:50 PM, you wrote:

Thank you for the reply. I started to comment first... but it was more
philosophy a mature and grown up, experienced man and I don't think
I have right to comment it.

Statistically if you have more users the probability of the system
survival of any architecture, philosophy or type is higher. People
learn, they're not fixed and if they at the beginning do not share
the philosophy of the system but they can use it - they may like it,
understand it and follow it and support later. Many people I asked
are not minding to help Gentoo getting better by turning on
feedback. If you remember - feedback worked well for Perl once and
many used it and Perl is very traditional.

It's like a chess game. You have the system in it's prime. There is
already one fork from Gentoo. There will be more. It's inevitable. You
have to understand that not all the developers share the same
philosophy - and it OK.
And they may fork Gentoo with time and pull half of the team to their
side.

When there is a competition between systems with equal philosophy the
only thing that stands between who is going to live and who is going
to die is the number of users.
The fight will focus not around philosophy or system but around gaining
user support. The competitor can build a better, more friendly system
sharing basically the same design and he will win it over.

To keep in power it's in your deepest interest to close the open gates that
invite competition while the power is in your hands. This is a failure
many grown up companies made they belive they're forever and gods. I could
share with you privately with several examples that prove that concept
wrong.

Your competitors will build basically the same system targeting the
same philosophy but more user oriented, friendly. User oriented - means
each user opinion matters.

There might be millions of users but each is treated like he is the only one.


PortageQOS is small step, it's not everything or main part of the
system, it's a just small contribution. But it will close the door and
you'll have another peaceful 8 years to rule.

What we need is a vote YES or NO. If you against it - vote NO. It's
perfectly normal, if there would be no NO there would be no need voting.


> Actually, in that regard it's very possible that gentoo's long planned
> and worked toward cvs-to-git conversion will help finally bust that 
> barrier for gentoo as well.  Time will tell I guess, but that's one more
> reason to try to help make it happen.




-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

Re: [gentoo-dev] Re: Portage QOS

[Chris Reffett]

On 01/09/2014 03:42 PM, Igor wrote:
> Hello Duncan,
> 
> Thursday, January 9, 2014, 9:59:50 PM, you wrote:
> 
> Thank you for the reply. I started to comment first... but it was more
> philosophy a mature and grown up, experienced man and I don't think
> I have right to comment it.
> 
> Statistically if you have more users the probability of the system
> survival of any architecture, philosophy or type is higher. People
> learn, they're not fixed and if they at the beginning do not share
> the philosophy of the system but they can use it - they may like it,
> understand it and follow it and support later. Many people I asked
> are not minding to help Gentoo getting better by turning on
> feedback. If you remember - feedback worked well for Perl once and
> many used it and Perl is very traditional.
> 
> It's like a chess game. You have the system in it's prime. There is
> already one fork from Gentoo. There will be more. It's inevitable. You
> have to understand that not all the developers share the same
> philosophy - and it OK.
> And they may fork Gentoo with time and pull half of the team to their
> side.
> 
> When there is a competition between systems with equal philosophy the
> only thing that stands between who is going to live and who is going
> to die is the number of users.
> The fight will focus not around philosophy or system but around gaining
> user support. The competitor can build a better, more friendly system
> sharing basically the same design and he will win it over.
> 
> To keep in power it's in your deepest interest to close the open gates that
> invite competition while the power is in your hands. This is a failure
> many grown up companies made they belive they're forever and gods. I could
> share with you privately with several examples that prove that concept
> wrong.
> 
> Your competitors will build basically the same system targeting the
> same philosophy but more user oriented, friendly. User oriented - means
> each user opinion matters.
> 
> There might be millions of users but each is treated like he is the only one.
> 
> 
> PortageQOS is small step, it's not everything or main part of the
> system, it's a just small contribution. But it will close the door and
> you'll have another peaceful 8 years to rule.
> 
Right here is the big problem: you're not looking at this from the
perspective of the average Gentoo developer. We don't care about market
share. We don't care whether we're on top for another few years. There
are several forks of Gentoo. I doubt most devs care about them. I
personally know that we're not going to compete with Debian, which has a
huge contributor, or Ubuntu or Red Hat, which have whole companies
behind them. You're selling this as if you're selling to a company which
wants to be on the top of the market and beating out competitors, and
that's not what we are. We are a source-based distro that requires some
effort from users, and people want that or they don't want it.
> What we need is a vote YES or NO. If you against it - vote NO. It's
> perfectly normal, if there would be no NO there would be no need voting.
> 
> 
>> Actually, in that regard it's very possible that gentoo's long planned
>> and worked toward cvs-to-git conversion will help finally bust that 
>> barrier for gentoo as well.  Time will tell I guess, but that's one more
>> reason to try to help make it happen.
> 
> 
> 
> 
Chris Reffett

Re: [gentoo-dev] Portage QOS

[heroxbd]

Igor <lanthruster@gmail.com> writes:

> The ebuilds have approximately the same time to install, the failure
> rate is about the same, emerge is getting slower.

I am curious about the slowness of emerge.

How about profile the portage and rewrite the time-crucial part in
C/C++, or ideally, borrowing the counterpart from paludis? How feasible
is that?

I guess the dep-tree calculation is the slowest part.

Re: [gentoo-dev] Portage QOS

[heroxbd]

Hey Igor,

Igor <lanthruster@gmail.com> writes:

> Jeroen, tell me how many users world wide do not prefer to upgrade Gentoo
> on automated basis? There are important servers, and there are many
> cases when after upgrade server stops. Do you remember that recent udev
> change? And there are many similar cases. Imagine that your server
> is running a reactor. So what would you prefer to keep it running the
> reactor as it did flawlessly for 8 years or launch an upgrade taking
> the risks to blast yourself?
>
> Many be it's not only me, but somebody else who is thinking the same?
> Are you sure that the majority of Gentoo users are indulged in
> paranoid automated upgrade and then spending time fixing damage
> that upgrade did?
>
> Do you have a car? Why you don't change EVERY detail in your car on a new
> version on daily basis automatically?
>
> Why don't you change car as soon as a new version is released? Why not
> changing the new mouse, new keyboard, new monitor, new supply daily as
> soon as there is a new version?
>
> Not to mention that you can change daily appearances.

IMHO, the bleeding-edgeness and stability form a balance. We cannot
achieve both. Taking RHEL for example, it uses ancient software for the
sake of stability. Gentoo is way off the other extreme.

For the udev change, the upstream has been doing evil and eudev is not
introduced as the default for Gentoo (yet).

New software breaks things, and security-updated old software needs
extra care: That's the fundamental problem we couldn't circumvent.

Cheers,
Benda

Re: [gentoo-dev] Portage QOS

[Patrick Lauer]

On 01/10/2014 08:16 AM, heroxbd@gentoo.org wrote:
> Igor <lanthruster@gmail.com> writes:
> 
>> The ebuilds have approximately the same time to install, the failure
>> rate is about the same, emerge is getting slower.
> 
> I am curious about the slowness of emerge.
> 
> How about profile the portage and rewrite the time-crucial part in
> C/C++, or ideally, borrowing the counterpart from paludis? How feasible
> is that?

Last I checked paludis wasn't faster - on average portage was a few
percents faster.

For python things you really want  python or C instead of C++...

So, what you wanted to ask was:
"Which parts of pkgcore can be migrated into portage?"

> I guess the dep-tree calculation is the slowest part.
Yes, it's doing lots of silly dynamic things (backtracking), and portage
codebase
on average is not designed for speed.

As a first step I would recommend profiling it and removing unneeded
stuff (do less work!), rewriting parts in C is a lot of work and not
needed for the first round of speedups.

Re: [gentoo-dev] Portage QOS

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

On Fri, 10 Jan 2014 09:16:47 +0900
heroxbd@gentoo.org wrote:

> Igor <lanthruster@gmail.com> writes:
> 
> > The ebuilds have approximately the same time to install, the failure
> > rate is about the same, emerge is getting slower.
> 
> I am curious about the slowness of emerge.

Try a --backtrack=0 approach, I no longer need to increase it. :)

> How about profile the portage and rewrite the time-crucial part in
> C/C++, or ideally, borrowing the counterpart from paludis? How
> feasible is that?

http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0.png
http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0-hot.png
http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-3.3-backtrack-0.png

(hot is the hotshot profiler, it internally checks on the line level
instead; 3.3's profiler is obstructed by module loading, no idea why)

> I guess the dep-tree calculation is the slowest part.

Affirmative.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Portage QOS

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 3042 bytes --]

On Fri, 10 Jan 2014 08:31:21 +0800
Patrick Lauer <patrick@gentoo.org> wrote:

> On 01/10/2014 08:16 AM, heroxbd@gentoo.org wrote:
> > Igor <lanthruster@gmail.com> writes:
> > 
> >> The ebuilds have approximately the same time to install, the
> >> failure rate is about the same, emerge is getting slower.
> > 
> > I am curious about the slowness of emerge.
> > 
> > How about profile the portage and rewrite the time-crucial part in
> > C/C++, or ideally, borrowing the counterpart from paludis? How
> > feasible is that?
> 
> Last I checked paludis wasn't faster - on average portage was a few
> percents faster.

Besides that, a different Python version can also differ; I've found
Python 3.3 to be faster for both emerge and repoman. Maybe PyPy can end
up even being faster than that, but that's another subjective story;
though introducing multiple threads in Portage would be really nice,
but the GIL sits in the way:

https://wiki.python.org/moin/GlobalInterpreterLock
http://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-why

> For python things you really want  python or C instead of C++...

Why is this a Python thing? What's the reason to exclude a language?

> So, what you wanted to ask was:
> "Which parts of pkgcore can be migrated into portage?"

Or rather: "What does it take to migrate parts of pkgcore into portage?"

> > I guess the dep-tree calculation is the slowest part.
> Yes, it's doing lots of silly dynamic things (backtracking),

Exactly, without backtracking (which I can live without) it takes
around half a minute as compared to a lot of minutes; when it started
to take almost half an hour it was time to completely nuke backtracking.

Now I happily live without ever needing to enable backtracking again. :)

> and portage codebase on average is not designed for speed.

Yes, inspecting the profiler results has me worried; though I find half
a minute acceptable for the amount of packages that are involved on my
system, so, I'm less concerned but it is still interesting that there
is the possibility to do things faster. It's just some work to actually
get it to do that, which requires much more dedication, time and
knowledge than doing an occasional commit.

> As a first step I would recommend profiling it and removing unneeded
> stuff (do less work!), rewriting parts in C is a lot of work and not
> needed for the first round of speedups.

See my other mail <20140110020218.0f6244d5@TOMWIJ-GENTOO> for recent
profiling images; we should indeed start with dealing with the low
hanging fruit (there are quite some 0-2% speed improvements possible),
as that can get us to speed it up faster than trying to deal with some
algorithmic complexity that needs far more insight to redesign, let
alone to properly re-implement it.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Portage QOS

[Patrick McLean]

On Fri, 10 Jan 2014 02:19:03 +0100
Tom Wijsman <TomWij@gentoo.org> wrote:

> On Fri, 10 Jan 2014 08:31:21 +0800
> Patrick Lauer <patrick@gentoo.org> wrote:
> 
> > On 01/10/2014 08:16 AM, heroxbd@gentoo.org wrote:
> > > Igor <lanthruster@gmail.com> writes:
> > > 
> > >> The ebuilds have approximately the same time to install, the
> > >> failure rate is about the same, emerge is getting slower.
> > > 
> > > I am curious about the slowness of emerge.
> > > 
> > > How about profile the portage and rewrite the time-crucial part in
> > > C/C++, or ideally, borrowing the counterpart from paludis? How
> > > feasible is that?
> > 
> > Last I checked paludis wasn't faster - on average portage was a few
> > percents faster.
> 

> > For python things you really want  python or C instead of C++...
> 
> Why is this a Python thing? What's the reason to exclude a language?
> 
> > So, what you wanted to ask was:
> > "Which parts of pkgcore can be migrated into portage?"
> 
> Or rather: "What does it take to migrate parts of pkgcore into
> portage?"

Why not just switch to using pkgcore as the default package manager.
radhermit has been doing a lot of work lately getting EAPI 5 support
added, and generally fixing bugs etc.

Since we no longer have anyone intimately familiar with the
portage code, we should just switch to a more modern and readable
system. Rather than having random people trying to learn the convoluted
portage code, let's concentrate on getting pkgcore to a point where
we can replace portage with it.

> > > I guess the dep-tree calculation is the slowest part.
> > Yes, it's doing lots of silly dynamic things (backtracking),
> 
> Exactly, without backtracking (which I can live without) it takes
> around half a minute as compared to a lot of minutes; when it started
> to take almost half an hour it was time to completely nuke
> backtracking.
> 
> Now I happily live without ever needing to enable backtracking
> again. :)
> 
> > and portage codebase on average is not designed for speed.
> 
> Yes, inspecting the profiler results has me worried; though I find
> half a minute acceptable for the amount of packages that are involved
> on my system, so, I'm less concerned but it is still interesting that
> there is the possibility to do things faster. It's just some work to
> actually get it to do that, which requires much more dedication, time
> and knowledge than doing an occasional commit.
> 
> > As a first step I would recommend profiling it and removing unneeded
> > stuff (do less work!), rewriting parts in C is a lot of work and not
> > needed for the first round of speedups.
> 
> See my other mail <20140110020218.0f6244d5@TOMWIJ-GENTOO> for recent
> profiling images; we should indeed start with dealing with the low
> hanging fruit (there are quite some 0-2% speed improvements possible),
> as that can get us to speed it up faster than trying to deal with some
> algorithmic complexity that needs far more insight to redesign, let
> alone to properly re-implement it.
> 

Re: [gentoo-dev] Portage QOS

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2515 bytes --]

On Thu, 9 Jan 2014 17:52:16 -0800
Patrick McLean <chutzpah@gentoo.org> wrote:

> On Fri, 10 Jan 2014 02:19:03 +0100
> Tom Wijsman <TomWij@gentoo.org> wrote:
> 
> > Or rather: "What does it take to migrate parts of pkgcore into
> > portage?"
> 
> Why not just switch to using pkgcore as the default package manager.

Has anyone switched to pkgcore already?

It needs to work well and be wide tested before it can become a default.

> radhermit has been doing a lot of work lately getting EAPI 5 support
> added, and generally fixing bugs etc.

Are we there yet?

The thing about pkgcore is that it is perceived as running behind; and
with the lack of interest in it due to that, it seems that having it
will take some time before it lifts off the ground. Don't get me wrong,
it eventually will if we pursue it; but, when? It can take time.

> Since we no longer have anyone intimately familiar with the
> portage code, we should just switch to a more modern and readable
> system.

Agreed, but we also should keep Portage alive and kicking until then.

> Rather than having random people trying to learn the
> convoluted portage code, let's concentrate on getting pkgcore to a
> point where we can replace portage with it.

Either way, people need to learn something; and if we can't guarantee
the short term future of Portage before pkgcore becomes usable, the
long term future could rather be out of reach before you know it.

In the short term we should focus on Portage, but in the long term we
should indeed look at one or another replacement; and indeed does
pkgcore soon appear to be the most interesting option here.

To satisfy QA and Portage teams at the same time; I'm going to start
digging into repoman soon as there are 80+ bugs open for it, so in the
short term (days / weeks), I have no plans for pkgcore myself.

From there on I plan to do a software re-engineering style inspection on
the Portage base to learn and understand its convoluted structure; at
that point we can make more educated choices as to which way to go
forward, but until then ... let's just fix as much bugs as time permits.

Don't get me wrong; pkgcore is great, but it takes time for attention
to shift to it as Portage's slowly runs into more legacy code problems.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Portage QOS

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

On Thu, 2014-01-09 at 17:52 -0800, Patrick McLean wrote:
> On Fri, 10 Jan 2014 02:19:03 +0100
> Tom Wijsman <TomWij@gentoo.org> wrote:
> 
> > On Fri, 10 Jan 2014 08:31:21 +0800
> > Patrick Lauer <patrick@gentoo.org> wrote:
> > 
> > > On 01/10/2014 08:16 AM, heroxbd@gentoo.org wrote:
> > > Last I checked paludis wasn't faster - on average portage was a few
> > > percents faster.
> > 
> 
> > > For python things you really want  python or C instead of C++...
> > 
> > Why is this a Python thing? What's the reason to exclude a language?
> > 
> > > So, what you wanted to ask was:
> > > "Which parts of pkgcore can be migrated into portage?"
> > 
> > Or rather: "What does it take to migrate parts of pkgcore into
> > portage?"
> 
> Why not just switch to using pkgcore as the default package manager.
> radhermit has been doing a lot of work lately getting EAPI 5 support
> added, and generally fixing bugs etc.
> 
> Since we no longer have anyone intimately familiar with the
> portage code, we should just switch to a more modern and readable
> system. Rather than having random people trying to learn the convoluted
> portage code, let's concentrate on getting pkgcore to a point where
> we can replace portage with it.
> 

Patrick  If you wish to be a part of getting pkgcore's eapi 5 fully
working your welcome to start helping out.  I can set you up with access
in the main google code repo.  Radhermit has been working mostly out of
his github account, updating the main repo.

I would very much love to get pkgcore fully functional in eapi 5.

It's speed runs circles around portage and paludis.

I have not been able to help radhermit out as yet, I have been trying to
get some other project cleaned up before delving deeper into pkgcore
code.  I have just recently taken over lead of portage (an interim
position) due to Zac stepping down and away.  So I have that added to my
todo list at the moment too. 

-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 620 bytes --]

Re: [gentoo-dev] Portage QOS

[heroxbd]

Patrick Lauer <patrick@gentoo.org> writes:

> For python things you really want  python or C instead of C++...

Well, we have boost-python to do python extensions in C++. And yes,
introducing boost as a dependency to portage is not cool.

>> I guess the dep-tree calculation is the slowest part.
> Yes, it's doing lots of silly dynamic things (backtracking), and
> portage codebase on average is not designed for speed.
>
> As a first step I would recommend profiling it and removing unneeded
> stuff (do less work!), rewriting parts in C is a lot of work and not
> needed for the first round of speedups.

Cython[1] can be used to generate C code quickly to avoid spending to much
time coding in C.

1. http://www.cython.org

Re: [gentoo-dev] Portage QOS

[heroxbd]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

Tom Wijsman <TomWij@gentoo.org> writes:

>> I am curious about the slowness of emerge.
>
> Try a --backtrack=0 approach, I no longer need to increase it. :)

on a random box:

time emerge --backtrack=0 -pe @world
[...]
real    0m30.016s
user    0m29.268s
sys     0m0.704s

time emerge -pe @world
[...]
real    0m35.037s
user    0m30.824s
sys     0m1.136s

not a big difference?

>> How about profile the portage and rewrite the time-crucial part in
>> C/C++, or ideally, borrowing the counterpart from paludis? How
>> feasible is that?
>
> http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0.png
> http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-2.7-backtrack-0-hot.png
> http://dev.gentoo.org/~tomwij/files/portage-2.2.7-python-3.3-backtrack-0.png
>
> (hot is the hotshot profiler, it internally checks on the line level
> instead; 3.3's profiler is obstructed by module loading, no idea why)

Great! That's what I am looking for.

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: [gentoo-dev] Re: Portage QOS

[Igor]

[-- Attachment #1: Type: text/plain, Size: 1572 bytes --]

Hello Chris,

Friday, January 10, 2014, 1:08:39 AM, you wrote:

> Right here is the big problem: you're not looking at this from the
> perspective of the average Gentoo developer. We don't care about market
> share. We don't care whether we're on top for another few years. There
> are several forks of Gentoo. I doubt most devs care about them. I
> personally know that we're not going to compete with Debian, which has a
> huge contributor, or Ubuntu or Red Hat, which have whole companies
> behind them. You're selling this as if you're selling to a company which
> wants to be on the top of the market and beating out competitors, and
> that's not what we are. We are a source-based distro that requires some
> effort from users, and people want that or they don't want it.
>> What we need is a vote YES or NO. If you against it - vote NO. It's
>> perfectly normal, if there would be no NO there would be no need voting.

Thank you for you opinion. 

The competition in open source world is much harder than with
commercial software. 3 commercial systems share 96% of the users.

1,6% of the users are shared by 296 Linux distros

You may think that you're outside this rules but the competition is natural 
on the planet and Gentoo is certainly competing weather you want it or not.

Competition was long before a human foot stood on the ground for the first 
time :-) And suddenly there is no competition in Linux world - do you really 
belive in it?


-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 2351 bytes --]

Re: [gentoo-dev] Portage QOS

[Igor]

Hello Heroxbd,

Friday, January 10, 2014, 4:16:47 AM, you wrote:

>> The ebuilds have approximately the same time to install, the failure
>> rate is about the same, emerge is getting slower.

> I am curious about the slowness of emerge.

> How about profile the portage and rewrite the time-crucial part in
> C/C++, or ideally, borrowing the counterpart from paludis? How feasible
> is that?

> I guess the dep-tree calculation is the slowest part.

If you had the PortageQOS you would see what change slowed down
Portage. And the problem could have already been fixed.
You could make fast and correct decisions. True, it could be possible
that some parts of the portage need to be rewritten. May be Python was
the wrong decision, may be it's better to use C++.

(Yes, I expect to be
condemned over here before I'm banned, a sacrifice for the better Gentoo
somebody had to make).

Do you remember how many problems portage had with Python? It's like
Gentoo is for Python not for anything else.

Why not to get rid of Python at all. What is so great in Python that
Gentoo exists for the sake of it?

And when next thing is introduced - you can see how it works
world wide. 

On some older PC the new portage works for 4-6 minutes before it FAILS.

If the portage is going to be a little bit smarter again - you would need a
new hardware to run it.

And nobody cares, of course it's better to hide don't know or run from
problems than know about them and fix them.

300 devs, are NOT ABLE to make portage fast in 8 years.

-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

Re: [gentoo-dev] Re: Portage QOS

[René Neumann]

Am 10.01.2014 13:10, schrieb Igor:
> Hello Chris,
> 
> Friday, January 10, 2014, 1:08:39 AM, you wrote:
> 
>> Right here is the big problem: you're not looking at this from the
>> perspective of the average Gentoo developer. We don't care about market
>> share. We don't care whether we're on top for another few years. There
>> are several forks of Gentoo. I doubt most devs care about them. I
>> personally know that we're not going to compete with Debian, which has a
>> huge contributor, or Ubuntu or Red Hat, which have whole companies
>> behind them. You're selling this as if you're selling to a company which
>> wants to be on the top of the market and beating out competitors, and
>> that's not what we are. We are a source-based distro that requires some
>> effort from users, and people want that or they don't want it.
>>> What we need is a vote YES or NO. If you against it - vote NO. It's
>>> perfectly normal, if there would be no NO there would be no need voting.
> 
> Thank you for you opinion. 
> 
> The competition in open source world is much harder than with
> commercial software. 3 commercial systems share 96% of the users.
> 
> 1,6% of the users are shared by 296 Linux distros
> 
> You may think that you're outside this rules but the competition is natural 
> on the planet and Gentoo is certainly competing weather you want it or not.
> 
> Competition was long before a human foot stood on the ground for the first 
> time :-) And suddenly there is no competition in Linux world - do you really 
> belive in it?

There is no really competition for the non-enterprise systems. It is a
co-existance (one might even call it some kind of symbiosis).

Please take your business nonsense somewhere else. Honestly, you sound
like some suit with his powerpoint slides talking about buzz-words like
'competitors', 'keeping in power', 'QoS' …

There were also numerous threads already about the very same topic. Most
came to the conclusions that
a) Gentoo is not dying
b) the numbers used as arguments are inaccurate at best (how do you
count 'Gentoo users'? And do you want users or machines? And what with
persons using different systems)

- René

Re: [gentoo-dev] Portage QOS

[Igor]

Hello Heroxbd,

Friday, January 10, 2014, 4:16:47 AM, you wrote:

>> The ebuilds have approximately the same time to install, the failure
>> rate is about the same, emerge is getting slower.

> I am curious about the slowness of emerge.

> How about profile the portage and rewrite the time-crucial part in
> C/C++, or ideally, borrowing the counterpart from paludis? How feasible
> is that?

> I guess the dep-tree calculation is the slowest part.

And to think about it - Python is a slow big snake. And Gentoo is the
fastest of penguins.

So why do we send Gentoo for food riding on Python? If it were death
we send Gentoo for then I would choose Python but food?

Isn't it making them both slow and food isn't coming? 



-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

Re: [gentoo-dev] Portage QOS

[René Neumann]

Am 10.01.2014 13:23, schrieb Igor:
> You could make fast and correct decisions. 

There is no such thing as the single correct decision. Management people
often think there is, but this is because management people often have
no clue what they are talking about.

> 
> Why not to get rid of Python at all. What is so great in Python that
> Gentoo exists for the sake of it?
>

What is so bad with Python? Also keep in mind: A bad/wrong algorithm
does not get better just because you change the implementation language.

> 
> 300 devs, are NOT ABLE to make portage fast in 8 years.
> 

You obviously have no idea how Gentoo works.

- René

Re: [gentoo-dev] Portage QOS

[Patrick Lauer]

On 01/10/2014 08:30 PM, Igor wrote:
> Hello Heroxbd,
> 
> Friday, January 10, 2014, 4:16:47 AM, you wrote:
> 
>>> The ebuilds have approximately the same time to install, the failure
>>> rate is about the same, emerge is getting slower.
> 
>> I am curious about the slowness of emerge.
> 
>> How about profile the portage and rewrite the time-crucial part in
>> C/C++, or ideally, borrowing the counterpart from paludis? How feasible
>> is that?
> 
>> I guess the dep-tree calculation is the slowest part.
> 
> And to think about it - Python is a slow big snake. And Gentoo is the
> fastest of penguins.

No, Python isn't slow.

Bad code is bad. You can write bad code in any language.
> 
> So why do we send Gentoo for food riding on Python? If it were death
> we send Gentoo for then I would choose Python but food?

I'm finding it very hard to stay polite, because ... honestly?

You have no idea what you're talking about.

If you want things to change - hire a few of us fulltime to work on
things, and you'll get the change you want.
Until then there's no need to point out that we are lacking manpower to
do large-scale changes, because that's been a constant in most
opensource projects since the 1960s.

Less talking, more doing - provide patches and stop polluting our
mailing list with your madness.

Re: [gentoo-dev] Portage QOS

[Igor]

Hello Heroxbd,

Friday, January 10, 2014, 4:27:00 AM, you wrote:

> IMHO, the bleeding-edgeness and stability form a balance. We cannot
> achieve both. Taking RHEL for example, it uses ancient software for the
> sake of stability. Gentoo is way off the other extreme.

> For the udev change, the upstream has been doing evil and eudev is not
> introduced as the default for Gentoo (yet).

> New software breaks things, and security-updated old software needs
> extra care: That's the fundamental problem we couldn't circumvent.

True, it's fundamentally impossible for Gentoo to get rid of all the
problems with ebuilds.

What I offer is to make the response and self-assessment on Gentoo
changes automated and fast. Then it will be getting better by itself.
The rate of experience Dev is attaining will jump several times up and
the level drudgery will decrease in the same proportion.

And it's perfectly in Gentoo style. This is the fastest penguin,
erroneously married to a snake because the marriage looked easy from
the first but now it's getting harder and harder.

We have the Penguin but forgot to add neurons to his skin.

He is poked but doesn't know about that and doesn't respond
because he can't feel.

-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

Re: [gentoo-dev] Re: Portage QOS

[Igor]

Hello René,

Friday, January 10, 2014, 4:26:03 PM, you wrote:


>> You may think that you're outside this rules but the competition is natural 
>> on the planet and Gentoo is certainly competing weather you want it or not.

>> Competition was long before a human foot stood on the ground for the first 
>> time :-) And suddenly there is no competition in Linux world - do you really 
>> belive in it?

> There is no really competition for the non-enterprise systems. It is a
> co-existance (one might even call it some kind of symbiosis).

> Please take your business nonsense somewhere else. Honestly, you sound
> like some suit with his powerpoint slides talking about buzz-words like
> 'competitors', 'keeping in power', 'QoS' …

> There were also numerous threads already about the very same topic. Most
> came to the conclusions that
> a) Gentoo is not dying
> b) the numbers used as arguments are inaccurate at best (how do you
> count 'Gentoo users'? And do you want users or machines? And what with
> persons using different systems)

You're living right not in competition. If you're on an island
you compete with animals for food and water. If you're in a condo - hell, 
you know how many on this planet WISH to live in your house right now and 
what stops them from doing that?

And you belive that you're outside competition. It looks unreal.
Gentoo is in competition with other distros - it's real and happens
right now.

Are you absolutely sure that in the condition when nobody knows how
Portage works we may go that far as saying we have a healthy Penguin?

-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

Re: [gentoo-dev] Re: Portage QOS

[René Neumann]

Am 10.01.2014 13:52, schrieb Igor:
> And you belive that you're outside competition. It looks unreal.
> Gentoo is in competition with other distros - it's real and happens
> right now.

Again, just because this "science" called 'Economics' believes,
everything is in competition, does not change reality.

> Are you absolutely sure that in the condition when nobody knows how
> Portage works we may go that far as saying we have a healthy Penguin?

If I want to know whether or not a penguin is healthy, I'd ask my mum
(she's a vet).

- René

Re: [gentoo-dev] Portage QOS

[Igor]

Hello Patrick,

Friday, January 10, 2014, 4:39:59 PM, you wrote:

> No, Python isn't slow.
> Bad code is bad. You can write bad code in any language.

Are you sure? Take a look here:

http://benchmarksgame.alioth.debian.org/u32q/benchmark.php?test=all&lang=python3&lang2=gpp&data=u32q

of course these stats are all so fake, and you may not belive them.

I've been using C/C++ since school it's fast, even bad code is working fast.

I WOULD NEVER BELIVE PYTHON IS AS FAST AS C++ with math algorithms
that do calculate staff and not call functions from pre-complied
objects written in C/C++.

It's crazy that you're even trying to state it. Take a look at what
Python is producing in disasm and then look at it in G++.

>> So why do we send Gentoo for food riding on Python? If it were death
>> we send Gentoo for then I would choose Python but food?

> I'm finding it very hard to stay polite, because ... honestly?
> You have no idea what you're talking about.

Or vice versa.

> If you want things to change - hire a few of us fulltime to work on
> things, and you'll get the change you want.
> Until then there's no need to point out that we are lacking manpower to
> do large-scale changes, because that's been a constant in most
> opensource projects since the 1960s.

> Less talking, more doing - provide patches and stop polluting our
> mailing list with your madness.

See tge subject of this letter. The whole point of this conversation is that
I offered to design it and program it and offered HARDWARE for it but we
can't get to the point because it's not clear for everyone if we need it.

If high command not needing it it will find means to kill it and I'm
very busy, really very busy - can't afford to spend that much time on
something not useful.

We're in the middle of negotiations here.

-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

Re: [gentoo-dev] Portage QOS

[Igor]

Hello Patrick,

Friday, January 10, 2014, 4:39:59 PM, you wrote:

> Bad code is bad. You can write bad code in any language.

BTW Perl is faster than Python too. 

Try writing quick sort in Perl, Ptyhon and G++ 

then dump the memory. 

And watch the miracle.

-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

Re: [gentoo-dev] Portage QOS

[René Neumann]

Am 10.01.2014 14:05, schrieb Igor:
> Hello Patrick,
> 
> Friday, January 10, 2014, 4:39:59 PM, you wrote:
> 
>> No, Python isn't slow.
>> Bad code is bad. You can write bad code in any language.
> 
> Are you sure? Take a look here:
> 
> http://benchmarksgame.alioth.debian.org/u32q/benchmark.php?test=all&lang=python3&lang2=gpp&data=u32q
> 
> of course these stats are all so fake, and you may not belive them.
> 
> I've been using C/C++ since school it's fast, even bad code is working fast.
> 
> I WOULD NEVER BELIVE PYTHON IS AS FAST AS C++ with math algorithms

You do realize, that we are not doing math (read: number crunching) here?

And again: What is needed is streamlining the algorithms (discussion on
that already started as far as I could notice). An algorithm in O(n³) is
always¹ worse than O(n). The constant factor added by the language
difference is of no interest.

> It's crazy that you're even trying to state it. Take a look at what
> Python is producing in disasm and then look at it in G++.

For a larger project, it often is more important to have readable and
maintable code opposed to getting the last bit of performance. And
Python is _far_ more readable and concise than C or C++ (imho). Due to
lack of typechecking, I'm not so sure when it comes to maintainability
though (there are test suites of course).

- René

¹ For a large enough n.

Re: [gentoo-dev] Portage QOS

[Rich Freeman]

On Fri, Jan 10, 2014 at 7:41 AM, Igor <lanthruster@gmail.com> wrote:
> What I offer is to make the response and self-assessment on Gentoo
> changes automated and fast. Then it will be getting better by itself.
> The rate of experience Dev is attaining will jump several times up and
> the level drudgery will decrease in the same proportion.

Sounds great.  Don't let commentary stand in your way.  Just post a
link to your code/patches when you have something to share, or if
you're looking for contributors to join you.  That's pretty-much how
every substantial improvement to any FOSS project starts out.

Rich

Re: [gentoo-dev] Portage QOS

[Rich Freeman]

On Fri, Jan 10, 2014 at 8:10 AM, Igor <lanthruster@gmail.com> wrote:
> Hello Patrick,
>
> Friday, January 10, 2014, 4:39:59 PM, you wrote:
>
>> Bad code is bad. You can write bad code in any language.
>
> BTW Perl is faster than Python too.
>
> Try writing quick sort in Perl, Ptyhon and G++
>
> then dump the memory.
>
> And watch the miracle.

I think you're missing the point.

If I ask somebody who knows nothing about algorithms to sort a list in
Python they're going to use foo.sort().  If I ask somebody who knows
nothing about algorithms to sort a list in C they're going to write a
bubble sort, and it will be WAY slower for anything more than a dozen
elements.

Honestly, you're writing as if you're talking to a bunch of people who
don't know anything about how computers work, and the reality is that
you'll be hard-pressed to find an audience more familiar with
compilers/toolchains/linkers/etc just about anywhere.

If you have the right algorithm nobody is arguing that it will run
faster if compiled from correctly-written C.  The problem is that
right now we don't have the right algorithm, and we're likely to get a
lot further with fixing that faster in a language like python than in
C.

But, nobody is opposing the work - there are two alternative package
managers for Gentoo today, and one of them is full-featured.  Neither
are written in python.

Rich

Re: [gentoo-dev] Portage QOS

[heroxbd]

Hi Igor,

Igor <lanthruster@gmail.com> writes:

> I've been using C/C++ since school it's fast, even bad code is working fast.
>
> I WOULD NEVER BELIVE PYTHON IS AS FAST AS C++ with math algorithms
> that do calculate staff and not call functions from pre-complied
> objects written in C/C++.
>
> It's crazy that you're even trying to state it. Take a look at what
> Python is producing in disasm and then look at it in G++.
>
>>> So why do we send Gentoo for food riding on Python? If it were death
>>> we send Gentoo for then I would choose Python but food?
>
>> I'm finding it very hard to stay polite, because ... honestly?
>> You have no idea what you're talking about.
>
> Or vice versa.
>
>> If you want things to change - hire a few of us fulltime to work on
>> things, and you'll get the change you want.
>> Until then there's no need to point out that we are lacking manpower to
>> do large-scale changes, because that's been a constant in most
>> opensource projects since the 1960s.
>
>> Less talking, more doing - provide patches and stop polluting our
>> mailing list with your madness.
>
> See tge subject of this letter. The whole point of this conversation is that
> I offered to design it and program it and offered HARDWARE for it but we
> can't get to the point because it's not clear for everyone if we need it.
>
> If high command not needing it it will find means to kill it and I'm
> very busy, really very busy - can't afford to spend that much time on
> something not useful.

I appreciate your intension to make Gentoo better, at the same time I
could not share the view with you.

Here is the checklist for you:

1. I got a brilliant idea! goto 2
2. Can I realize it by myself?
   Y, goto 6; N goto 3
3. Can I convince someone to do it for me?
   Y, goto 6; N goto 4
4. Can I hire someone to do it for me?
   Y, goto 6; N goto 5
5. Nothing could be achieved :(
6. After rounds of hard work, it comes true :D

Seems that you got stuck at 5. Sorry if you are not willing to adjust
your attitude, your subsequent will be ignored.

Some of the details of your proposal do interest me, such as an
automatic build.log / emerge --info uploader. Supporting mail reply to
bugzilla will even be cooler (which can be used to upload build.log and
emerge --info of course).

Cheers,
Benda

Re: [gentoo-dev] Portage QOS

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

On Fri, 10 Jan 2014 18:10:05 +0900
heroxbd@gentoo.org wrote:

> Tom Wijsman <TomWij@gentoo.org> writes:
> 
> >> I am curious about the slowness of emerge.
> >
> > Try a --backtrack=0 approach, I no longer need to increase it. :)
> 
> on a random box:
> 
> time emerge --backtrack=0 -pe @world
> [...]
> real    0m30.016s
> user    0m29.268s
> sys     0m0.704s
> 
> time emerge -pe @world
> [...]
> real    0m35.037s
> user    0m30.824s
> sys     0m1.136s
> 
> not a big difference?

Well, it has to actually backtrack to make a difference; if there's no
need to, it won't differ. But when there is need to, it takes longer.

That's why it has been described as dynamic; you are either affected by
a lot of backtracking or you don't, seems you are on the lucky side now.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Portage QOS

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 1933 bytes --]

On Fri, 10 Jan 2014 09:02:46 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> On Fri, Jan 10, 2014 at 8:10 AM, Igor <lanthruster@gmail.com> wrote:
> If I ask somebody who knows nothing about algorithms to sort a list in
> Python they're going to use foo.sort().  If I ask somebody who knows
> nothing about algorithms to sort a list in C they're going to write a
> bubble sort, and it will be WAY slower for anything more than a dozen
> elements.

This assumes that the person doesn't do it manually in Python and
doesn't make use of already implemented functionality (eg. qsort) in C,
which jumps out as the first Google result; generalizations like these
are subjective, because it's just your view and thoughts of reality.

> Honestly, you're writing as if you're talking to a bunch of people who
> don't know anything about how computers work, and the reality is that
> you'll be hard-pressed to find an audience more familiar with
> compilers/toolchains/linkers/etc just about anywhere.

Indeed, a lot of us are CompSci students have that algorithmic
complexity drilled in since the first year. If we need performance,
we'll put it to great use; an occasional prototype, not so much.

> If you have the right algorithm nobody is arguing that it will run
> faster if compiled from correctly-written C.  The problem is that
> right now we don't have the right algorithm, and we're likely to get a
> lot further with fixing that faster in a language like python than in
> C.

Actually, language doesn't even matter here; just push that power off
button 'n get back to paper, then fix it in even faster pseudo code. :)

(Well, unless you type pseudo code faster than you write it down ... :P)

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Portage QOS

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2911 bytes --]

On Fri, 10 Jan 2014 16:52:12 +0400
Igor <lanthruster@gmail.com> wrote:

> You're living right not in competition.If you're on an island
> you compete with animals for food and water. If you're in a condo -
> hell, you know how many on this planet WISH to live in your house
> right now and what stops them from doing that?
> 
> And you belive that you're outside competition. It looks unreal.
> Gentoo is in competition with other distros - it's real and happens
> right now.

That's the marketing view on it; but now take a look from the customer
view, why am I and you on Gentoo or one of its derivatives and not
on some other distribution out there. Exactly, Gentoo provides some
things that are hard to find in other distributions; patching the
source, having more choice, remove features you don't need, etc...

Customers are looking for that; my thought exactly when I was looking
for Gentoo was "I want to be able to easily manipulate what I use",
Gentoo's ability to just patch the source code is great for that.

I'd be scared of going to a binary distro, the distros where you are
source based are quite limited; if I'd quit Gentoo, I'm for sure I'll
either be on a Gentoo-derived distribution or build something similar.

The former makes me still be on Gentoo, or at least its idea; the
latter might be quite an effort that'll take quite some time too get
going, but I'll know for sure if that ever jumped my mind that I would
miss the Portage tree and so on.

So, other non-derived distros don't really look like competitors to me;
they might satisfy a set of our customers, but then you can just as
well realize that those customers have no real need for Gentoo and were
bound to leave sooner or later anyway.

As for derived distros; forking is great, right?

We are no longer using the old solid wheels that break or the old cars
that fail to continue to drive after a bit of usage; no, we are using
well designed, well implemented, well tested wheels and cars that
continue to drive for months to years.

And as an example, we also have ideas like those small cars which only
fit one person; who knows, that idea never catches on in public. But at
least we've satisfied the needs of a few people out there; not everyone
wants to have such a car, just a few persons that are in need for it.

> Are you absolutely sure that in the condition when nobody knows how
> Portage works we may go that far as saying we have a healthy Penguin?

Reverse engineering a car will not say anything about its health;
fixing its bugs, testing it and supporting it however will.

Otherwise we could claim solid wheels and old cars to still be healthy.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Portage QOS

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 57 bytes --]

please do not use html on the public mailing lists
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Portage QOS

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

On Fri, 10 Jan 2014 08:31:21 +0800
Patrick Lauer <patrick@gentoo.org> wrote:
> On 01/10/2014 08:16 AM, heroxbd@gentoo.org wrote:
> > Igor <lanthruster@gmail.com> writes:
> > 
> >> The ebuilds have approximately the same time to install, the
> >> failure rate is about the same, emerge is getting slower.
> > 
> > I am curious about the slowness of emerge.
> > 
> > How about profile the portage and rewrite the time-crucial part in
> > C/C++, or ideally, borrowing the counterpart from paludis? How
> > feasible is that?
> 
> Last I checked paludis wasn't faster - on average portage was a few
> percents faster.

Your benchmark was comparing uncached behaviour, where bash is the slow
part and which users don't see. You were also not comparing like with
like -- any benchmarks of this nature should be taken with a heavy
pinch of salt, since Portage with everything turned on does less
validation that Paludis does with everything turned off...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Portage QOS

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 180 bytes --]

On Fri, 10 Jan 2014 09:16:47 +0900
heroxbd@gentoo.org wrote:
> or ideally, borrowing the counterpart from paludis? How feasible is
> that?

It's not.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Portage QOS

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

On Thu, 9 Jan 2014 17:52:16 -0800
Patrick McLean <chutzpah@gentoo.org> wrote:
> Why not just switch to using pkgcore as the default package manager.
> radhermit has been doing a lot of work lately getting EAPI 5 support
> added, and generally fixing bugs etc.

Pkgcore is dead (see: still no EAPI 5 support). If you're really
thinking about switching, the way to go is to write a client for
Paludis which mimics Portage's UI. But the politics make switching from
Portage to anything a lost cause.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Portage QOS

[Greg KH]

On Fri, Jan 10, 2014 at 04:10:18PM +0400, Igor wrote:
> Hello Chris,
> 
> Friday, January 10, 2014, 1:08:39 AM, you wrote:
> 
> > Right here is the big problem: you're not looking at this from the
> > perspective of the average Gentoo developer. We don't care about market
> > share. We don't care whether we're on top for another few years. There
> > are several forks of Gentoo. I doubt most devs care about them. I
> > personally know that we're not going to compete with Debian, which has a
> > huge contributor, or Ubuntu or Red Hat, which have whole companies
> > behind them. You're selling this as if you're selling to a company which
> > wants to be on the top of the market and beating out competitors, and
> > that's not what we are. We are a source-based distro that requires some
> > effort from users, and people want that or they don't want it.
> >> What we need is a vote YES or NO. If you against it - vote NO. It's
> >> perfectly normal, if there would be no NO there would be no need voting.
> 
> Thank you for you opinion. 
> 
> The competition in open source world is much harder than with
> commercial software. 3 commercial systems share 96% of the users.

Really?  What market?  Last I looked, more Linux systems were being run
on processors than any other single kernel.

Don't confuse desktops for "all of computing".

> 1,6% of the users are shared by 296 Linux distros

Really?  And what is the number of total users you are talking about
here?

> You may think that you're outside this rules but the competition is natural 
> on the planet and Gentoo is certainly competing weather you want it or not.

No, we aren't competing with anyone, please realize that this is not how
Linux distros and the ecosystem works at all.

Remember, Red Hat is Gentoo's engineering team :)

good luck,

greg k-h

Re: [gentoo-dev] Portage QOS

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

On Fri, 10 Jan 2014 14:18:24 +0100
René Neumann <lists@necoro.eu> wrote:
> And again: What is needed is streamlining the algorithms (discussion
> on that already started as far as I could notice). An algorithm in
> O(n³) is always¹ worse than O(n). The constant factor added by the

Full dependency resolution is NP-hard... If you really wanted to
streamline the algorithms, you'd change the inputs slightly. Most of
the complexity of doing a decent job of dependency resolution comes from
dealing with crap input.

> ¹ For a large enough n.

...which could be larger than the number of atoms in the universe.
There are real world cases where an algorithm has an O(n^3) step that
takes a day, and a O(2^n) step that takes a second for most inputs. You
shouldn't be using O() to make performance comparisons.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Portage QOS

[René Neumann]

Am 10.01.2014 19:19, schrieb Ciaran McCreesh:
> On Fri, 10 Jan 2014 14:18:24 +0100
> René Neumann <lists@necoro.eu> wrote:
>> And again: What is needed is streamlining the algorithms (discussion
>> on that already started as far as I could notice). An algorithm in
>> O(n³) is always¹ worse than O(n). The constant factor added by the
> 
> Full dependency resolution is NP-hard... 

Though NP-hard does not necessarily mean 'unfeasible in practice'.

> If you really wanted to
> streamline the algorithms, you'd change the inputs slightly. Most of
> the complexity of doing a decent job of dependency resolution comes from
> dealing with crap input.

My intention really was not to tell anybody how the depres algorithms
should be improved. I just wanted to make a point against the 'Python is
the root of all the bad performance'.

>> ¹ For a large enough n.
> 
> ...which could be larger than the number of atoms in the universe.
> There are real world cases where an algorithm has an O(n^3) step that
> takes a day, and a O(2^n) step that takes a second for most inputs. You
> shouldn't be using O() to make performance comparisons.
> 

Point taken. I should've use 'in general' instead of 'always'. What I
had in mind when writing this were more smaller problems, where a good
algorithm exists but people use their own naïve implementation/data
structure.

- René

[gentoo-dev] Re: Portage QOS

[Duncan]

Chris Reffett posted on Thu, 09 Jan 2014 16:08:39 -0500 as excerpted:

>> To keep in power it's in your deepest interest to close the open gates
>> that invite competition while the power is in your hands.

>> PortageQOS is small step, it's not everything or main part of the
>> system, it's a just small contribution. But it will close the door and
>> you'll have another peaceful 8 years to rule.
>> 
> Right here is the big problem: you're not looking at this from the
> perspective of the average Gentoo developer. We don't care about market
> share. We don't care whether we're on top for another few years. There
> are several forks of Gentoo. I doubt most devs care about them.

Agreed with Chris.

@ Igor:

Closed... open.  You clearly don't seem to get it.

There's a reason it's called "open source", contrasting favorably with 
"closed source".  You're seriously pushing deliberately closing the door 
on people's choice?  That's typical closed-source methology, and what 
many FLOSS folks would consider the antithesis of the entire reason they 
do what they do.  And that's your sales point, to a community-based open-
source-based distro, not to some closed source company like MS?

If Gentoo dies, well, rest in peace, Gentoo, it was good while it lasted, 
but people moved on to something that suited their needs better.

That's the whole point.  If people are more comfortable with a binary 
distro, there's lots of them available, and many gentoo users and devs 
are even happy to take some time to listen to what a user needs and make 
the best recommendation they can about a distro that fits that need.  If 
people want a distro that's near as expert level and with near the choice 
of gentoo, but don't want to /always/ have to build /everything/ from 
source, Arch Linux is over that way, or if you want to stick with the 
general gentoo idea but want a few tweaks, Funtoo's over that way, and 
Sabayon is over there.  And if they want to go full-out and build 
/everything/ from source without gentoo's automated framework, well, 
Linux from Scratch is right over yonder too!

The thing is, we don't see this as a game where if those distros win, we 
lose, or if we win, they lose.  Instead, we're different players on the 
same team, and if we can helpfully direct users their way that are more 
comfortable with them than with us, great, and we'll hope they'll do the 
same with people who might be more comfortable here, too, but we're not 
making that a condition of our directing people we know will be more 
comfortable there to them.

Similarly, we're on the same team when it comes to patches.  No distro or 
their developers/maintainers want the *FULL* burden of supporting all 
those packages, and if Gentoo devs can find a Fedora or Debian patch that 
solves a problem we too have, or if we came up with a patch first that 
solves a problem they're having too, great, have at it!

And in all that, if Gentoo's former devs and users all end up on Arch or 
LFS instead, as I said, rest in peace, Gentoo, it was good having known 
you, but your time appears to have been up, and others took your place.

But you're talking deliberately closing doors and walling in users so 
they don't switch, instead of happily pointing them at distros they'd 
obviously be more comfortable with.  WTF is that sort of attitude doing 
here in free/libre and open source in the first place?  That's more the 
walled garden Apple or MS approach, not FLOSS.

Meanwhile, you might try googling Zynot.  That was one early, perhaps the 
first, Gentoo fork.  Such talk of cutthroat competition in a zero-sum 
game, of deliberately cutting off user options so they'd be forced to 
stick with you, of it can be us or them, not both, etc, was exactly the 
sort of thing they tried.  That was 2002/2003 or so.  While the events 
and acrimony surrounding that did ultimately drive Gentoo's founder 
(Daniel Robbins) elsewhere, Gentoo survived (thanks in part to drobbins' 
efforts to secure a good future for it even at heavy personal cost to 
himself and his family as he was already in the process of leaving).  
Gentoo's still here, but where is zynot today?

I remember back in early 2004 as I was researching my switch to gentoo, 
reading up on zynot, which was at that time still a going concern, and 
repeatedly asking myself as I read the essays from zynot's founder 
heavily criticizing gentoo and its founder, why can't he see what's 
happening, that every single thing he's negatively pointing at in gentoo 
and drobbins he and zynot are doing themselves in far greater measure, 
and why he was so stuck on closed source competitive techniques in an 
open source world.  His very essays, supposedly criticizing gentoo, 
instead ended up convincing me more than ever that gentoo was /exactly/ 
the right choice for me. =:^)

And... gentoo's still here, but where is zynot, today?  I think I made 
the right choice then, and I'm still convinced it was then and remains 
now, the right choice, for me, today. =:^)

But if gentoo dies as a result of following those policies, well, it 
dies, and I, and other gentoo users and devs, will find something else to 
replace it.  But gentoo didn't die when zynot was saying those things 
(ultimately, zynot did), and pardon me for saying so, but I don't see it 
dieing now, when you're saying them.  Instead, the risk of death is if we 
belief and follow them now, just as it was then.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Portage QOS

[heroxbd]

Duncan <1i5t5.duncan@cox.net> writes:

> Meanwhile, you might try googling Zynot.  That was one early, perhaps the 
> first, Gentoo fork.  Such talk of cutthroat competition in a zero-sum 
> game, of deliberately cutting off user options so they'd be forced to 
> stick with you, of it can be us or them, not both, etc, was exactly the 
> sort of thing they tried.  That was 2002/2003 or so.  While the events 
> and acrimony surrounding that did ultimately drive Gentoo's founder 
> (Daniel Robbins) elsewhere, Gentoo survived (thanks in part to drobbins' 
> efforts to secure a good future for it even at heavy personal cost to 
> himself and his family as he was already in the process of leaving).  
> Gentoo's still here, but where is zynot today?
>
> I remember back in early 2004 as I was researching my switch to gentoo, 
> reading up on zynot, which was at that time still a going concern, and 
> repeatedly asking myself as I read the essays from zynot's founder 
> heavily criticizing gentoo and its founder, why can't he see what's 
> happening, that every single thing he's negatively pointing at in gentoo 
> and drobbins he and zynot are doing themselves in far greater measure, 
> and why he was so stuck on closed source competitive techniques in an 
> open source world.  His very essays, supposedly criticizing gentoo, 
> instead ended up convincing me more than ever that gentoo was /exactly/ 
> the right choice for me. =:^)

Wow... What a history! I am educated. Thanks for sharing.

I've always been interested in my distro's history. The information
scatters here and there. It'll be nice if some senior/retired developers
write up a Gentoo history on wiki.g.o :)

Benda

[gentoo-dev] Re: Portage QOS

[Duncan]

heroxbd posted on Sat, 11 Jan 2014 07:36:57 +0900 as excerpted:

> Duncan <1i5t5.duncan@cox.net> writes:
> 
>> Meanwhile, you might try googling Zynot.  That was one early, perhaps
>> the first, Gentoo fork.
>>
>> I remember back in early 2004
> 
> Wow... What a history! I am educated. Thanks for sharing.
> 
> I've always been interested in my distro's history. The information
> scatters here and there. It'll be nice if some senior/retired developers
> write up a Gentoo history on wiki.g.o :)

FWIW, I did my research and ended up on gentoo after the split was 
basically done, tho zynot was still around at the time.  As such, I don't 
have a lot of personal experience with it, but it was still close enough 
that most of the gentoo devs of the time did have personal experience.

What I do know is that it was a very bitter experience for many, and most 
that lived thru it, like many survivors of a lot of particularly man-made 
tragedies, considered the experience something that they and gentoo had 
survived, and were /extremely/ glad it was over, but weren't much for 
talking about it.

At a safe historic distance of a decade in the past, perhaps some might 
talk about it now, but I'd guess for many, it's just not worth reliving, 
except, $deity forbid, should there be a danger of something similar 
occurring again.  Too many bitter recriminations.  Too many previous 
friends lost to the split...

But I was close enough time-wise to appreciate the seriousness and 
tragedy of the event, while not being part of it myself, so I don't have 
those old wounds to rip back open by talking about it.  Apologies to the 
long-time devs still here for whom I'm doing just that, but it /is/ 
history now, and as the saying goes, those who don't know history are 
bound to repeat it, something I'm absolutely sure NOBODY involved would 
want, so...

From what I understand, this guy /had/ been effectively drobbins' right-
hand-man for a time.  He had business connections and had been 
instrumental in parlaying some of them into gentoo sponsorships at a time 
when it was much younger and needed them, and he was a good PR guy.  The 
gentoo dev community was smaller and closer knit at the time, and many 
had considered this guy and the devs that ultimately left with him 
personal friends.  That made the hurt /much/ worse. =:^(

What I've always wondered is what the devs who went with him thought; how 
he persuaded them, /their/ side of the story.  I knew /his/ side of the 
story from reading his essays attacking gentoo and drobbins, and I knew 
at least enough about the gentoo side to be convinced that the gentoo 
side was where I should be, but coming in shortly after as I did, I never 
had any contact with or read anything from any of the devs that left with 
him, and I obviously didn't know them previously, so their side of the 
story, why he convinced them to go zynot (other than the obvious, that 
any persuasive argument must have /some/ element of truth), I'll never 
know.  Meanwhile, I'm /quite/ aware that my own view and recounting of 
the history I know is quite colored by my own position, and definitely 
/must/ suffer to some degree from the "victor rewriting history" 
phenomenon.  I'm sure if I had a better view of the picture as the devs 
who left for zynot saw it, that "people who left" view would be rather 
different, and regardless of whether I agreed with it or not, it would 
certainly color my own view and thus recounting of the facts as I am 
aware of them.  Worth keeping in mind...

Meanwhile, that /some/ bit of truth, AFAIK, revolved around the fact that 
while gentoo had settled on the GPLv2 for code and similarly free general 
documentation licenses, drobbins was apparently asking for copyright 
rights, with a policy of copyright everything gentoo, which drobbins held 
the rights to, with the ownership rights becoming the core of the fight.  
There had been some talk of some sort of a gaming distro (I'm fuzzy on 
the details), apparently drobbins' big idea, and as a base for embedded, 
this guy's big idea and ultimately zynot's target for funding, etc.  This 
guy accused drobbins of intending to do the gaming thing then take 
everything private.  As I wasn't there and am not drobbins, I can't say 
for sure what drobbins ultimate idea and motives were, but as I read this 
guy's essays, I kept shouting at the monitor, "But if he intended to go 
private and deprive other contributors of their just due, why GPLv2, not 
MIT/BSD, which would make that so much easier?"  Of course as we know 
from the MySQL/Sun/Oracle events, with all rights a company can still go 
private, using the GPL to maintain an unfair advantage over others who 
can't take it private because they don't have the copyrights, only the GPL 
version.  But even so, again as the MySQL/Oracle/MariaDB events, and the 
Sun/Oracle/OpenOffice/LibreOffice events as well demonstrate, if that's 
against the wishes of an already active and developed community, that 
community can and will take the free version it still has rights to use 
and run with it!

Meanwhile, from all I could see then and to the extent that I know 
anything of zynot to this day, that's EXACTLY what zynot tried to do, 
take advantage of the free-licensed gentoo work and extend it with their 
proprietary product.  Clear as anything else I've ever seen, it was the 
soot-covered pot looking in the mirror and believing it sees a kettle to 
call black!

That's enough old wounds I'm sure I've torn open for some, sorry.  But 
knowing that history explains QUITE A BIT of gentoo's internal politics 
to this day, so it's VERY worth knowing about for new devs who had no 
idea that was in gentoo's history.  Among other things, that definitely 
plays a part in why people are now encouraged to mark their work 
copyright gentoo if they have no strong feelings about it, but gentoo 
doesn't DEMAND it.  (Another factor is as greg-kh points out, due to 
employment contracts a lot of gentoo devs wouldn't be able to contribute 
and would have to resign, were a firm copyright rights assignment policy 
established.

It plays and even *STRONGER* role in gentoo's governing structure, both 
because drobbins took quite some care and personal legal expense to 
ensure a separate gentoo foundation with the assets, but *NOT* technical 
control, and in the very loose government structure, with little central 
control and individual devs having lots of rights that are rather 
difficult to strip, except by what ultimately amounts to overwhelming 
(but not necessarily unanimous) agreement (which does and has occurred 
when necessary, as some former devs who still follow this list can surely 
attest), should a case be appealed all the way thru council, etc.

And even tho there has been enough turnover that I don't believe the 
original devs have anything like enough power to directly maintain those 
rules, the original themes were strong enough to have set in motion a 
VERY strong culture of little central power and lots of individual dev 
independence, such that succeeding generations have continued to inherit 
that from their mentors and other devs that came before them.  Those 
original devs tended to attract others of like mind, and train them in 
the way, and that generation in turn did the same, such that while few 
newer devs really understand the history behind it, that comparatively 
weak central power and strong individual dev rights continue to this day.

And of course that same theme is playing in this thread.  Gentoo culture 
has an extremely strong emphasis on individual rights, including the 
right to choose one's own distribution, such that most gentoo devs (and 
users) will find the very idea of somehow deliberately closing off 
avenues of choice, restricting distro choice and the ability of users to 
leave if they feel so inclined, EXTREMELY repulsive.  Yes, to some extent 
the majority of the FLOSS community has a similar culture, but self-
evidently the typical dev in a typical corporate-sponsored distro isn't 
as likely to have the extreme, gut-level revulsion to centralized or 
corporate control of the distro, or to dev and user choice, that your 
typical gentooer dev is likely to have.

And actually, I'm glad this discussion has come up, since writing about 
it has given me new insights into things as well.  I obviously had all 
the factoids and history available before, but this has forced me to 
realize connections that I hadn't previously considered.

Wow!  I had thought that was just the way gentoo's culture was.  Now I 
understand a bit more about how its history shapes that, and /why/ 
gentoo's culture is the way it is.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Portage QOS

[Patrick Lauer]

On 01/11/2014 02:11 AM, Ciaran McCreesh wrote:
> On Fri, 10 Jan 2014 08:31:21 +0800
> Patrick Lauer <patrick@gentoo.org> wrote:
>> On 01/10/2014 08:16 AM, heroxbd@gentoo.org wrote:
>>> Igor <lanthruster@gmail.com> writes:
>>>
>>>> The ebuilds have approximately the same time to install, the
>>>> failure rate is about the same, emerge is getting slower.
>>>
>>> I am curious about the slowness of emerge.
>>>
>>> How about profile the portage and rewrite the time-crucial part in
>>> C/C++, or ideally, borrowing the counterpart from paludis? How
>>> feasible is that?
>>
>> Last I checked paludis wasn't faster - on average portage was a few
>> percents faster.
> 
> Your benchmark was comparing uncached behaviour, where bash is the slow
> part and which users don't see.
Wrong - even the cached cases was showing the same timing proportions.

And users see the uncached case whenever they use an overlay.

> You were also not comparing like with
> like -- any benchmarks of this nature should be taken with a heavy
> pinch of salt, since Portage with everything turned on does less
> validation that Paludis does with everything turned off...
> 
Not my problem, bad code is bad.

Re: [gentoo-dev] Portage QOS

[Naohiro Aota]

[-- Attachment #1: Type: text/plain, Size: 2780 bytes --]

Hi, Igor

LTHR <lanthruster@gmail.com> writes:

> Portage QOS
>
> Hi All,
>
> What do you think about implementing this:
>
> http://forums.gentoo.org/viewtopic.php?p=7477494
>
> I've system design in my head and could write it down with the
> implementation details.
> Then may be we could all review it and get to something we all agree
> upon then I could 
> try getting a team and implement it.
>
> Just a brief question - does anyone know how many ebuilds are
> assembled world 
> wide each second?

This is quite impressive for me. I'm one who have been thinking like
this. For the purpose, I started a Web service named GenTwoo on May
2011: http://gentwoo.elisp.net/?locale=en

First a GenTwoo user login with Twitter account to the service and
installs a tiny script [1] on their Gentoo box. Then each time they run
emerge, the script collect what package is emerged, if the emerge is
succeed or failed, its elog output, and its build.log (only if the
emerge failed). These information is sent to my server and tweeted
periodically with "#gentwoo" hash tag so that you can see your friends are
heating their computer :-). [2] 

You can also browse:

- What all GenTwoo users emerge recently
  http://gentwoo.elisp.net/emerges?locale=en
- What one user emerge recently
  http://gentwoo.elisp.net/users/naota344?locale=en
- How a emerge failed (click "error log" tab)
  http://gentwoo.elisp.net/emerges/644259?locale=en
- Recent popular packages
  http://gentwoo.elisp.net/poppackage?locale=en
- How long dose it take to emerge a package (and its average)
  http://gentwoo.elisp.net/packages/app-text/poppler/?locale=en#0.24.5

Since I'm not much advertising the project, there are not so many active
users (and they are mostly Japanese): there are only 54 users who ever
emerged since Dec 2013. I'm not sure my GenTwoo project completely suit
your demand, but there are already many emerge record on my server
(645112 emerge records since the service started, and 22129 emerges
since Dec 2013). So if you are interested, you can start "PortStat" or
"PortStatDEV" implementation immediately with the emerge data I have. Or
you can also join GenTwoo if you feel it's better to start from
scratch. I'd appreciate it if you would join GenTwoo project and improve
it together ;-) There's also on going project to rewrite GenTwoo into a
package stat [3]

Anyway, feel free to ask me GenTwoo's implementation detail if you are
interested in it. I think I can help you write your design idea if you
start from scratch with your idea.

[1] https://github.com/naota/gentwoo/blob/master/client/gentwoo.py
[2] https://twitter.com/search?q=%23gentwoo&f=realtime
[3] https://github.com/gentoo/GenTwoo-backend/

Regards,

[-- Attachment #2: Type: application/pgp-signature, Size: 835 bytes --]

Re: [gentoo-dev] Portage QOS

[Igor]

[-- Attachment #1: Type: text/plain, Size: 5212 bytes --]

Hello Naohiro,

Saturday, January 11, 2014, 7:00:58 PM, you wrote:

Thanks! I'll peek into it today. 

Emery Hemingway all so generously offered his help. 

What I plan to do is to share the system design with the people who thinks 
the same. I won't mind if either of us will implement it when it's 
partially ready or we all may contribute to the cause when it's ready. 

My personal goal is to help Gentoo a bit as it's free project I don't care much 
about propriety. 

Let's know each other a bit better. 

I'm good with C++/PERL/PHP/JS/HTML/FASTCGI, hardware, services, like Apache, MYSQL, bind. 
I've experience of designing systems that handle about 1 million hosts daily. 

The most problematic part in this project apart from a political task of getting it into the portage 
is going to be the load that it can handle. In our case the load is a killer, if this system is slow or 
unstable it will be deployed in portage.

I don't know how many users gentoo has world wide I have to design the system scalable, such systems 
can hold as many host as there is hardware/bandwidth available. 

Theoretically there might about 100 000 ebuild done each second world wide. The hardware that I have at 
the moment if we design everything right can handle about 500 failed and 1000 total|second not more.

But I hope there will be no more than 1000 ebuilds/second load at the beginning. 

Let's register a domain portageqos.org and create a mailgroup on it so we could communicate more easily.

It's all so possible that before we have PortageQOS ready somebody else will deploy the same and there will 
be no need in this project any more. These are risk we would all have to accept.

But it's highly unlikely that there are many people who are willing to implement PQOS service and who has 
enough experience and hardware to actually launch this project. 

If we ever launch we would need to deploy in phases. First phase - we need to deploy first 1000 systems 
say, whose name is starting from a-c then we watch how we're doing and how much load we brought. Then we 
go further with alphabet extending the number of connected hosts.

If we reach max load on the infrastructure we would have to be ready to extend it.


> LTHR <lanthruster@gmail.com> writes:

>> Portage QOS

>> Hi All,

>> What do you think about implementing this:

>> http://forums.gentoo.org/viewtopic.php?p=7477494

>> I've system design in my head and could write it down with the
>> implementation details.
>> Then may be we could all review it and get to something we all agree
>> upon then I could 
>> try getting a team and implement it.

>> Just a brief question - does anyone know how many ebuilds are
>> assembled world 
>> wide each second?

> This is quite impressive for me. I'm one who have been thinking like
> this. For the purpose, I started a Web service named GenTwoo on May
> 2011: http://gentwoo.elisp.net/?locale=en

> First a GenTwoo user login with Twitter account to the service and
> installs a tiny script [1] on their Gentoo box. Then each time they run
> emerge, the script collect what package is emerged, if the emerge is
> succeed or failed, its elog output, and its build.log (only if the
> emerge failed). These information is sent to my server and tweeted
> periodically with "#gentwoo" hash tag so that you can see your friends are
> heating their computer :-). [2] 

> You can also browse:

> - What all GenTwoo users emerge recently
>   http://gentwoo.elisp.net/emerges?locale=en
> - What one user emerge recently
>   http://gentwoo.elisp.net/users/naota344?locale=en
> - How a emerge failed (click "error log" tab)
>   http://gentwoo.elisp.net/emerges/644259?locale=en
> - Recent popular packages
>   http://gentwoo.elisp.net/poppackage?locale=en
> - How long dose it take to emerge a package (and its average)
>   http://gentwoo.elisp.net/packages/app-text/poppler/?locale=en#0.24.5

> Since I'm not much advertising the project, there are not so many active
> users (and they are mostly Japanese): there are only 54 users who ever
> emerged since Dec 2013. I'm not sure my GenTwoo project completely suit
> your demand, but there are already many emerge record on my server
> (645112 emerge records since the service started, and 22129 emerges
> since Dec 2013). So if you are interested, you can start "PortStat" or
> "PortStatDEV" implementation immediately with the emerge data I have. Or
> you can also join GenTwoo if you feel it's better to start from
> scratch. I'd appreciate it if you would join GenTwoo project and improve
> it together ;-) There's also on going project to rewrite GenTwoo into a
> package stat [3]

> Anyway, feel free to ask me GenTwoo's implementation detail if you are
> interested in it. I think I can help you write your design idea if you
> start from scratch with your idea.

> [1] https://github.com/naota/gentwoo/blob/master/client/gentwoo.py
> [2] https://twitter.com/search?q=%23gentwoo&f=realtime
> [3] https://github.com/gentoo/GenTwoo-backend/

> Regards,



-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 8827 bytes --]

[gentoo-dev] Re: Portage QOS

[Steven J. Long]

On Fri, Jan 10, 2014 Igor wrote:
> I've been using C/C++ since school it's fast, even bad code is working fast.
> 
> I WOULD NEVER BELIVE PYTHON IS AS FAST AS C++ with math algorithms
> that do calculate staff and not call functions from pre-complied
> objects written in C/C++.

I would never believe it til I first saw pkgcore in action 5 or 6 years ago.
There's no point criticising portage, since everyone knows it's a pig of a
project, that's never had a rewrite: which is why its lead developer stepped
back and wrote pkgcore. And sure, most of its speed comes from using an
optimised C backend: snakeoil.

I'm assuming you have a Gentoo box and can use eix or equery to find these.
If not, you need to reconsider what you're doing. I'm also assuming you are
going to try pkgcore so you are better-informed, even if its latest
incarnation is not ready for mass-release; after all you're a developer,
so can deal with that, right?

> Friday, January 10, 2014, Patrick wrote:
> >> So why do we send Gentoo for food riding on Python? If it were death
> >> we send Gentoo for then I would choose Python but food?
> 
> > I'm finding it very hard to stay polite, because ... honestly?
> > You have no idea what you're talking about.
> 
> Or vice versa.

It's wierd, you veer between corporatist dogma, and mildly hallucinogenic
metaphor. So let's just say I have no idea what you are talking about in
some of your emails, or rather no idea why you reach for those metaphors.

> > If you want things to change - hire a few of us fulltime to work on
> > things, and you'll get the change you want.
> > Until then there's no need to point out that we are lacking manpower to
> > do large-scale changes, because that's been a constant in most
> > opensource projects since the 1960s.
> 
> > Less talking, more doing - provide patches and stop polluting our
> > mailing list with your madness.
> 
> See tge subject of this letter. The whole point of this conversation is that
> I offered to design it and program it and offered HARDWARE for it but we
> can't get to the point because it's not clear for everyone if we need it.

You're coming at this from the angle of a commercial developer, and basically
no-one really cares about any of that. There's loads of hardware available,
for example, since so many Gentoo users are in fact net admins. We do care
about improving the distro, so by all means go ahead and implement something;
the Gentwo thing sounds like a perfect collaboration opportunity.

If you want to work on something in FLOSS, you do so for your own reasons:
they're what keep you doing it, even when you think no-one else cares.
Coming at the list with your "offer to design and program" is the wrong way
round: loads of people offer the same sort of thing (it comes up about once
a year or so, afaict) and most never deliver anything. Project ideas are
two-a-penny.

Show us a working project, or the basis of one, and everything moves: other
users who want the same thing will help you with it. Bug reports will come
in and you can start to see where things need improvement. Assuming it
fills a need.

> If high command not needing it it will find means to kill it and I'm
> very busy, really very busy - can't afford to spend that much time on
> something not useful.
> 
> We're in the middle of negotiations here.

No, you are not. As Duncan's history lesson pointed out, there is no "high
command." As Rich pointed out, if you want to implement something, go right
ahead and do it. Don't seek permission, since there isn't anyone to give
it. If you build it, and it is useful, they will use it ;)

Note you won't get anything out of that, beyond the reputation you appear
to wish to establish. Few people will thank you, though those that do will
make it all worthwhile; mostly what you'll get is more work. But the
bug reports will make your software better, and teach you an awful lot in
the process.

If you're looking for it to be an official project, there's no such thing;
only projects any Gentoo dev can have hosted, which does not make them
official by any means. You can look to get an ebuild into the portage tree,
and you can look to get infra to use your work (much harder.)

You're a *long* way away from even being able to suggest they look at
something, afaict. And even then it may not be something considered
essential to the functioning of the distro, but rather best left as an
external site. Or y'know some devs might take it up and run with it.
But you have to put the work in first.

In commercial terms, you have to deliver the prototype before any
discussion can begin. Until then, it's just vapourware, and with respect,
we've heard it all before.

In essence: just do it. And try to reduce the amount you post to the list;
it'll stop you getting snide remarks later on. Waiting a day or two between
posts is always advisable, and sticking to max two in any one day stops you
getting drawn into flame-wars.

Good luck :)

Regards,
steveL
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Portage QOS

[Igor]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

Hello Naohiro,

Saturday, January 11, 2014, 7:00:58 PM, you wrote:

This one was close.

It would be great to have your help. Can you if need be 
write a plugin for portage to send|receive portage data from|to
the servers? 

I'm not well with Python. We would need a plug-in for the 
portage to send stats and to receive back the feedback.

The goal to have that plugin inserted into the portage.

> Anyway, feel free to ask me GenTwoo's implementation detail if you are
> interested in it. I think I can help you write your design idea if you
> start from scratch with your idea.

> [1] https://github.com/naota/gentwoo/blob/master/client/gentwoo.py
> [2] https://twitter.com/search?q=%23gentwoo&f=realtime
> [3] https://github.com/gentoo/GenTwoo-backend/

> Regards,



-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 2198 bytes --]

Re: [gentoo-dev] Portage QOS

[Igor]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

Hello Naohiro,

Saturday, January 11, 2014, 7:00:58 PM, you wrote:

This one was close.

It would be great to have your help. Can you if need be 
write a plugin for portage to send|receive portage data from|to
the servers? 

I'm not well with Python. We would need a plug-in for the 
portage to send stats and to receive back the feedback.

The goal to have that plugin inserted into the portage.

> Anyway, feel free to ask me GenTwoo's implementation detail if you are
> interested in it. I think I can help you write your design idea if you
> start from scratch with your idea.

> [1] https://github.com/naota/gentwoo/blob/master/client/gentwoo.py
> [2] https://twitter.com/search?q=%23gentwoo&f=realtime
> [3] https://github.com/gentoo/GenTwoo-backend/

> Regards,



-- 
Best regards,
 Igor                            mailto:lanthruster@gmail.com

[-- Attachment #2: Type: text/html, Size: 2198 bytes --]