* Re: [gentoo-user] accelerate emerge
2006-02-27 16:05 ` Alexander Skwar
@ 2006-02-27 17:27 ` Daniel da Veiga
2006-02-27 17:39 ` Hans-Werner Hilse
2006-02-28 22:27 ` Walter Dnes
2 siblings, 0 replies; 18+ messages in thread
From: Daniel da Veiga @ 2006-02-27 17:27 UTC (permalink / raw
To: gentoo-user
On 2/27/06, Alexander Skwar <listen@alexander.skwar.name> wrote:
> El Nino wrote:
>
> > is there a way to accelerate the fetching part of emerge
> > by using prozilla or some other tool?
>
> I never quite understood the sense in those tools.
>
> Why should "prozilla or some other tool" make the
> download be faster? When I download something with
> wget, or watch emerge invoking wget, it's always
> maxing out the saturation of the line.
>
Well, if you have a high bandwidth line, like 8MB ADSL for example,
the host providing the download usually limit your download speed
reserving some band to other requests. If you could "teach" portage to
fetch pieces (like some other tools do) from different mirrors till
you saturate your line, your download would be considerably faster
(all your band would be used).
So, lets say I can use a program and tell it to use the GENTOO_MIRRORS
variable to download various pieces of the same file (reaching the
mirror servers limit and after a while your band limit), this would
ensure you would always use the max band you can... I think I saw some
tool over the net that could do that... Maybe wget itself....
/me go read the man pages...
--
Daniel da Veiga
Computer Operator - RS - Brazil
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCM/IT/P/O d-? s:- a? C++$ UBLA++ P+ L++ E--- W+++$ N o+ K- w O M- V-
PS PE Y PGP- t+ 5 X+++ R+* tv b+ DI+++ D+ G+ e h+ r+ y++
------END GEEK CODE BLOCK------
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-user] accelerate emerge
2006-02-27 16:05 ` Alexander Skwar
2006-02-27 17:27 ` Daniel da Veiga
@ 2006-02-27 17:39 ` Hans-Werner Hilse
2006-02-27 18:32 ` Dave Nebinger
2006-02-28 22:27 ` Walter Dnes
2 siblings, 1 reply; 18+ messages in thread
From: Hans-Werner Hilse @ 2006-02-27 17:39 UTC (permalink / raw
To: gentoo-user
Hi,
On Mon, 27 Feb 2006 17:05:39 +0100 Alexander Skwar
<listen@alexander.skwar.name> wrote:
> Why should "prozilla or some other tool" make the
> download be faster? When I download something with
> wget, or watch emerge invoking wget, it's always
> maxing out the saturation of the line.
On my 1Gig line on my workstation at work it's usually _not_ saturizing
the line. But I decided that it's not very polite to use a parallel
fetching tool under these circumstances...
-hwh
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-user] accelerate emerge
2006-02-27 17:39 ` Hans-Werner Hilse
@ 2006-02-27 18:32 ` Dave Nebinger
0 siblings, 0 replies; 18+ messages in thread
From: Dave Nebinger @ 2006-02-27 18:32 UTC (permalink / raw
To: gentoo-user
Hans-Werner Hilse wrote:
>> Why should "prozilla or some other tool" make the
>> download be faster? When I download something with
>> wget, or watch emerge invoking wget, it's always
>> maxing out the saturation of the line.
>
> On my 1Gig line on my workstation at work it's usually _not_ saturizing
> the line. But I decided that it's not very polite to use a parallel
> fetching tool under these circumstances...
I would bet that has more to do with traffic shaping on your connection
to the external world than anything to do with the local bandwidth, in
which case you could probably parallel all you want w/o improving
download performance.
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-user] accelerate emerge
2006-02-27 16:05 ` Alexander Skwar
2006-02-27 17:27 ` Daniel da Veiga
2006-02-27 17:39 ` Hans-Werner Hilse
@ 2006-02-28 22:27 ` Walter Dnes
2006-02-28 22:55 ` Iain Buchanan
2006-02-28 23:42 ` Boyd Stephen Smith Jr.
2 siblings, 2 replies; 18+ messages in thread
From: Walter Dnes @ 2006-02-28 22:27 UTC (permalink / raw
To: gentoo-user
On Mon, Feb 27, 2006 at 05:05:39PM +0100, Alexander Skwar wrote
> El Nino wrote:
>
> > is there a way to accelerate the fetching part of emerge
> > by using prozilla or some other tool?
>
> I never quite understood the sense in those tools.
>
> Why should "prozilla or some other tool" make the
> download be faster? When I download something with
> wget, or watch emerge invoking wget, it's always
> maxing out the saturation of the line.
I have a different interpretation. Assume I'm doing an
emerge --deep --update --world
I've set my ADSL router-modem to log off after 15 minutes. The
sequence of events is something like...
1) emerge wants to download a package, so it attempts to connect to a
server, taking several seconds to wake up my ADSL router-modem
2) emerge downloads a big package, taking a few minutes to do so
3) emerge spends the next half hour building the big package, during
which time the modem-router logs off
4) emerge finally finishes building the package, and wants to work on
the next one... GOTO 1
I could run a short script
#!/bin/bash
emerge --deep --update --world --fetchonly
emerge --deep --update --world
...but I'd like to get an emerge going on the 1st package as soon as
it's finished downloading, whilst having the downloads of all the other
packages continue in a separate thread. When the 1st build is finished,
check whether the 2nd package has been downloaded. If not, wait. Then
build the 2nd package... etc, etc.
The best way to describe it is as a --fetchonly emerge that launches a
separate emerge as each individual package is finished downloading. The
"build" emerges should be serialized, i.e. only one build running at a
time, because a package may depend on the immediately preceding package.
--
Walter Dnes <waltdnes@waltdnes.org> In linux /sbin/init is Job #1
My musings on technology and security at http://tech_sec.blog.ca
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread* Re: [gentoo-user] accelerate emerge
2006-02-28 22:27 ` Walter Dnes
@ 2006-02-28 22:55 ` Iain Buchanan
2006-02-28 23:25 ` Mike Owen
2006-03-01 3:17 ` Michael A. Smith
2006-02-28 23:42 ` Boyd Stephen Smith Jr.
1 sibling, 2 replies; 18+ messages in thread
From: Iain Buchanan @ 2006-02-28 22:55 UTC (permalink / raw
To: gentoo-user
On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
>
> I have a different interpretation.
I assume you know about FEATURES="parallel-fetch"?
--
Iain Buchanan <iain at netspace dot net dot au>
"Elves and Dragons!" I says to him. "Cabbages and potatoes are better
for you and me."
-- J. R. R. Tolkien
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-user] accelerate emerge
2006-02-28 22:55 ` Iain Buchanan
@ 2006-02-28 23:25 ` Mike Owen
2006-02-28 23:42 ` Boyd Stephen Smith Jr.
2006-02-28 23:50 ` Iain Buchanan
2006-03-01 3:17 ` Michael A. Smith
1 sibling, 2 replies; 18+ messages in thread
From: Mike Owen @ 2006-02-28 23:25 UTC (permalink / raw
To: gentoo-user
On 2/28/06, Iain Buchanan <iaindb@netspace.net.au> wrote:
> On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> >
> > I have a different interpretation.
>
> I assume you know about FEATURES="parallel-fetch"?
It's probably not the best idea to recommend ~arch versions of portage.
Mike
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-user] accelerate emerge
2006-02-28 23:25 ` Mike Owen
@ 2006-02-28 23:42 ` Boyd Stephen Smith Jr.
2006-02-28 23:50 ` Iain Buchanan
1 sibling, 0 replies; 18+ messages in thread
From: Boyd Stephen Smith Jr. @ 2006-02-28 23:42 UTC (permalink / raw
To: gentoo-user
On Tuesday 28 February 2006 17:25, "Mike Owen" <kyphros@gmail.com> wrote
about 'Re: [gentoo-user] accelerate emerge':
> On 2/28/06, Iain Buchanan <iaindb@netspace.net.au> wrote:
> > On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> > > I have a different interpretation.
> >
> > I assume you know about FEATURES="parallel-fetch"?
>
> It's probably not the best idea to recommend ~arch versions of portage.
I will recommend going ~arch for portage, I've done so for over a year.
That said, there's no need to go ~arch just for this feature. See my other
post.
--
"If there's one thing we've established over the years,
it's that the vast majority of our users don't have the slightest
clue what's best for them in terms of package stability."
-- Gentoo Developer Ciaran McCreesh
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-user] accelerate emerge
2006-02-28 23:25 ` Mike Owen
2006-02-28 23:42 ` Boyd Stephen Smith Jr.
@ 2006-02-28 23:50 ` Iain Buchanan
1 sibling, 0 replies; 18+ messages in thread
From: Iain Buchanan @ 2006-02-28 23:50 UTC (permalink / raw
To: gentoo-user
On Tue, 2006-02-28 at 15:25 -0800, Mike Owen wrote:
> On 2/28/06, Iain Buchanan <iaindb@netspace.net.au> wrote:
> > On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> > >
> > > I have a different interpretation.
> >
> > I assume you know about FEATURES="parallel-fetch"?
>
> It's probably not the best idea to recommend ~arch versions of portage.
Why not? Its a great feature. My (two) systems have been running
entirely ~x86 ever since I installed them (years ago), and I've never
had any real problems. Only rarely do I have to recompile, or
downgrade, but usually it's all handled by portage anyway.
Standard disclaimer applies: just because it works now, doesn't mean it
will work in the future; not for production / isolated servers, etc.
Note that from the comments before you modify ~arch:
"... '~arch' is a superset of 'arch' which includes the unstable, in
testing, packages ... 'Broken' packages will not be added to testing ...
IF YOU ARE UNSURE OF YOUR ARCH, OR THE IMPLICATIONS, DO NOT MODIFY
[ACCEPT_KEYWORDS]".
:)
--
Iain Buchanan <iain at netspace dot net dot au>
The use of money is all the advantage there is to having money.
-- B. Franklin
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-user] accelerate emerge
2006-02-28 22:55 ` Iain Buchanan
2006-02-28 23:25 ` Mike Owen
@ 2006-03-01 3:17 ` Michael A. Smith
2006-03-01 4:11 ` Vladimir G. Ivanovic
1 sibling, 1 reply; 18+ messages in thread
From: Michael A. Smith @ 2006-03-01 3:17 UTC (permalink / raw
To: gentoo-user
Iain Buchanan wrote:
> On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
>
>> I have a different interpretation.
>
>
> I assume you know about FEATURES="parallel-fetch"?
How long has this been around? I couldn't find any mention of it in
the man files. Where is it documented?
Thanks,
Mike
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-user] accelerate emerge
2006-03-01 3:17 ` Michael A. Smith
@ 2006-03-01 4:11 ` Vladimir G. Ivanovic
0 siblings, 0 replies; 18+ messages in thread
From: Vladimir G. Ivanovic @ 2006-03-01 4:11 UTC (permalink / raw
To: gentoo-user
It's new in the latest testing version of emerge.
--- Vladimir
On Tue, 2006-02-28 at 22:17 -0500, Michael A. Smith wrote:
> Iain Buchanan wrote:
> > On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> >
> >> I have a different interpretation.
> >
> >
> > I assume you know about FEATURES="parallel-fetch"?
> How long has this been around? I couldn't find any mention of it in
> the man files. Where is it documented?
>
> Thanks,
> Mike
--
Vladimir G. Ivanovic <vladimir@acm.org>
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [gentoo-user] accelerate emerge
2006-02-28 22:27 ` Walter Dnes
2006-02-28 22:55 ` Iain Buchanan
@ 2006-02-28 23:42 ` Boyd Stephen Smith Jr.
1 sibling, 0 replies; 18+ messages in thread
From: Boyd Stephen Smith Jr. @ 2006-02-28 23:42 UTC (permalink / raw
To: gentoo-user
On Tuesday 28 February 2006 16:27, "Walter Dnes" <waltdnes@waltdnes.org>
wrote about 'Re: [gentoo-user] accelerate emerge':
> #!/bin/bash
> emerge --deep --update --world --fetchonly
> emerge --deep --update --world
>
> ...but I'd like to get an emerge going on the 1st package as soon as
> it's finished downloading, whilst having the downloads of all the other
> packages continue in a separate thread.
Try adding '&' to your first line to start it as a background process. I
did this for months. MAKE SURE emerge --info SHOWS THAT YOU ARE USING THE
DISTLOCKS FEATURE.
It's a separate process instead of a separate thread, but it should do what
you want. You may also want to throw a '> /dev/null 2>&-' before the '&',
but that depends on your sensibilities.
--
"If there's one thing we've established over the years,
it's that the vast majority of our users don't have the slightest
clue what's best for them in terms of package stability."
-- Gentoo Developer Ciaran McCreesh
--
gentoo-user@gentoo.org mailing list
^ permalink raw reply [flat|nested] 18+ messages in thread