Hello Naohiro,
Saturday, January 11, 2014, 7:00:58 PM, you wrote:
Thanks! I'll peek into it today.
Emery Hemingway all so generously offered his help.
What I plan to do is to share the system design with the people who thinks
the same. I won't mind if either of us will implement it when it's
partially ready or we all may contribute to the cause when it's ready.
My personal goal is to help Gentoo a bit as it's free project I don't care much
about propriety.
Let's know each other a bit better.
I'm good with C++/PERL/PHP/JS/HTML/FASTCGI, hardware, services, like Apache, MYSQL, bind.
I've experience of designing systems that handle about 1 million hosts daily.
The most problematic part in this project apart from a political task of getting it into the portage
is going to be the load that it can handle. In our case the load is a killer, if this system is slow or
unstable it will be deployed in portage.
I don't know how many users gentoo has world wide I have to design the system scalable, such systems
can hold as many host as there is hardware/bandwidth available.
Theoretically there might about 100 000 ebuild done each second world wide. The hardware that I have at
the moment if we design everything right can handle about 500 failed and 1000 total|second not more.
But I hope there will be no more than 1000 ebuilds/second load at the beginning.
Let's register a domain portageqos.org and create a mailgroup on it so we could communicate more easily.
It's all so possible that before we have PortageQOS ready somebody else will deploy the same and there will
be no need in this project any more. These are risk we would all have to accept.
But it's highly unlikely that there are many people who are willing to implement PQOS service and who has
enough experience and hardware to actually launch this project.
If we ever launch we would need to deploy in phases. First phase - we need to deploy first 1000 systems
say, whose name is starting from a-c then we watch how we're doing and how much load we brought. Then we
go further with alphabet extending the number of connected hosts.
If we reach max load on the infrastructure we would have to be ready to extend it.
> LTHR <lanthruster@gmail.com> writes:
>> Portage QOS
>> Hi All,
>> What do you think about implementing this:
>> http://forums.gentoo.org/viewtopic.php?p=7477494
>> I've system design in my head and could write it down with the
>> implementation details.
>> Then may be we could all review it and get to something we all agree
>> upon then I could
>> try getting a team and implement it.
>> Just a brief question - does anyone know how many ebuilds are
>> assembled world
>> wide each second?
> This is quite impressive for me. I'm one who have been thinking like
> this. For the purpose, I started a Web service named GenTwoo on May
> 2011: http://gentwoo.elisp.net/?locale=en
> First a GenTwoo user login with Twitter account to the service and
> installs a tiny script [1] on their Gentoo box. Then each time they run
> emerge, the script collect what package is emerged, if the emerge is
> succeed or failed, its elog output, and its build.log (only if the
> emerge failed). These information is sent to my server and tweeted
> periodically with "#gentwoo" hash tag so that you can see your friends are
> heating their computer :-). [2]
> You can also browse:
> - What all GenTwoo users emerge recently
> http://gentwoo.elisp.net/emerges?locale=en
> - What one user emerge recently
> http://gentwoo.elisp.net/users/naota344?locale=en
> - How a emerge failed (click "error log" tab)
> http://gentwoo.elisp.net/emerges/644259?locale=en
> - Recent popular packages
> http://gentwoo.elisp.net/poppackage?locale=en
> - How long dose it take to emerge a package (and its average)
> http://gentwoo.elisp.net/packages/app-text/poppler/?locale=en#0.24.5
> Since I'm not much advertising the project, there are not so many active
> users (and they are mostly Japanese): there are only 54 users who ever
> emerged since Dec 2013. I'm not sure my GenTwoo project completely suit
> your demand, but there are already many emerge record on my server
> (645112 emerge records since the service started, and 22129 emerges
> since Dec 2013). So if you are interested, you can start "PortStat" or
> "PortStatDEV" implementation immediately with the emerge data I have. Or
> you can also join GenTwoo if you feel it's better to start from
> scratch. I'd appreciate it if you would join GenTwoo project and improve
> it together ;-) There's also on going project to rewrite GenTwoo into a
> package stat [3]
> Anyway, feel free to ask me GenTwoo's implementation detail if you are
> interested in it. I think I can help you write your design idea if you
> start from scratch with your idea.
> [1] https://github.com/naota/gentwoo/blob/master/client/gentwoo.py
> [2] https://twitter.com/search?q=%23gentwoo&f=realtime
> [3] https://github.com/gentoo/GenTwoo-backend/
> Regards,
--
Best regards,
Igor mailto:lanthruster@gmail.com