[gentoo-user] accelerate emerge

[gentoo-user] accelerate emerge

[El Nino]

AybOwan!

  is there a way to accelerate the fetching part of emerge
by using prozilla or some other tool?

--
...
"The future lies ahead."
 _______________________
< Have you mooed today? >
 ----------------------------------------
        \    ^__^
         \   (oo) \_______
             (__) \             )\/\
                    | |-----w   |
                    | |        | |

2.6.15-gentoo-r1-sinhalese-jan201
(((o)))~--~--~--~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Proud to be a Sinhalese.
SINHALESE ARE GENIUSES OF IRRIGATION
http://easyweb.easynet.co.uk/~sydney/sinhales.htm

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Boris Fersing]

2006/2/27, El Nino <adaptit@gmail.com>:
> AybOwan!
>
>   is there a way to accelerate the fetching part of emerge
> by using prozilla or some other tool?
>
Hi,

if you want to use prozilla, just change the  FETCHCOMMAND and
RESUMECOMMAND in /etc/make.conf

There is some examples in /etc/make.conf.example, but not for prozilla ...

HTH.

Boris

PS: With portage-2.1_pre* you can use the parallel fetch, just add
"parallel-fetch" in the FEATURES variable ...
> ...
> "The future lies ahead."
>  _______________________
> < Have you mooed today? >
>  ----------------------------------------
>         \    ^__^
>          \   (oo) \_______
>              (__) \             )\/\
>                     | |-----w   |
>                     | |        | |
>
> 2.6.15-gentoo-r1-sinhalese-jan201
> (((o)))~--~--~--~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Proud to be a Sinhalese.
> SINHALESE ARE GENIUSES OF IRRIGATION
> http://easyweb.easynet.co.uk/~sydney/sinhales.htm
>
> --
> gentoo-user@gentoo.org mailing list
>
>
>


--
Quiconque me parle de Dieu en veut à ma bourse ou à ma liberté.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Alexander Skwar]

El Nino wrote:

>   is there a way to accelerate the fetching part of emerge
> by using prozilla or some other tool?

I never quite understood the sense in those tools.

Why should "prozilla or some other tool" make the
download be faster? When I download something with
wget, or watch emerge invoking wget, it's always
maxing out the saturation of the line.

Alexander Skwar
-- 
Doubt isn't the opposite of faith; it is an element of faith.
		-- Paul Tillich, German theologian.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[El Nino]

On 2/27/06, Boris Fersing <kernelsensei@gentoo.org> wrote:
> 2006/2/27, El Nino <adaptit@gmail.com>:
> > AybOwan!
> >
> >   is there a way to accelerate the fetching part of emerge
> > by using prozilla or some other tool?
> >
> Hi,
>
> if you want to use prozilla, just change the  FETCHCOMMAND and
> RESUMECOMMAND in /etc/make.conf
>
> There is some examples in /etc/make.conf.example, but not for prozilla ...
>
> HTH.
>
> Boris
>
> PS: With portage-2.1_pre* you can use the parallel fetch, just add
> "parallel-fetch" in the FEATURES variable ...

how can i get to know all these portage features?

> > ...
> > "The future lies ahead."
> >  _______________________
> > < Have you mooed today? >
> >  ----------------------------------------
> >         \    ^__^
> >          \   (oo) \_______
> >              (__) \             )\/\
> >                     | |-----w   |
> >                     | |        | |
> >
> > 2.6.15-gentoo-r1-sinhalese-jan201
> > (((o)))~--~--~--~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Proud to be a Sinhalese.
> > SINHALESE ARE GENIUSES OF IRRIGATION
> > http://easyweb.easynet.co.uk/~sydney/sinhales.htm
> >
> > --
> > gentoo-user@gentoo.org mailing list
> >
> >
> >
>
>
> --
> Quiconque me parle de Dieu en veut à ma bourse ou à ma liberté.
>
> --
> gentoo-user@gentoo.org mailing list
>
>


--
...
"The future lies ahead."
 _______________________
< Have you mooed today? >
 ----------------------------------------
        \    ^__^
         \   (oo) \_______
             (__) \             )\/\
                    | |-----w   |
                    | |        | |

2.6.15-gentoo-r1-sinhalese-jan201
(((o)))~--~--~--~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Proud to be a Sinhalese.
SINHALESE ARE GENIUSES OF IRRIGATION
http://easyweb.easynet.co.uk/~sydney/sinhales.htm

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Daniel da Veiga]

On 2/27/06, Alexander Skwar <listen@alexander.skwar.name> wrote:
> El Nino wrote:
>
> >   is there a way to accelerate the fetching part of emerge
> > by using prozilla or some other tool?
>
> I never quite understood the sense in those tools.
>
> Why should "prozilla or some other tool" make the
> download be faster? When I download something with
> wget, or watch emerge invoking wget, it's always
> maxing out the saturation of the line.
>

Well, if you have a high bandwidth line, like 8MB ADSL for example,
the host providing the download usually limit your download speed
reserving some band to other requests. If you could "teach" portage to
fetch pieces (like some other tools do) from different mirrors till
you saturate your line, your download would be considerably faster
(all your band would be used).

So, lets say I can use a program and tell it to use the GENTOO_MIRRORS
variable to download various pieces of the same file (reaching the
mirror servers limit and after a while your band limit), this would
ensure you would always use the max band you can... I think I saw some
tool over the net that could do that... Maybe wget itself....

/me go read the man pages...

--
Daniel da Veiga
Computer Operator - RS - Brazil
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCM/IT/P/O d-? s:- a? C++$ UBLA++ P+ L++ E--- W+++$ N o+ K- w O M- V-
PS PE Y PGP- t+ 5 X+++ R+* tv b+ DI+++ D+ G+ e h+ r+ y++
------END GEEK CODE BLOCK------

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Hans-Werner Hilse]

Hi,

On Mon, 27 Feb 2006 17:05:39 +0100 Alexander Skwar
<listen@alexander.skwar.name> wrote:

> Why should "prozilla or some other tool" make the
> download be faster? When I download something with
> wget, or watch emerge invoking wget, it's always
> maxing out the saturation of the line.

On my 1Gig line on my workstation at work it's usually _not_ saturizing
the line. But I decided that it's not very polite to use a parallel
fetching tool under these circumstances...

-hwh
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Dave Nebinger]

Hans-Werner Hilse wrote:
>> Why should "prozilla or some other tool" make the
>> download be faster? When I download something with
>> wget, or watch emerge invoking wget, it's always
>> maxing out the saturation of the line.
> 
> On my 1Gig line on my workstation at work it's usually _not_ saturizing
> the line. But I decided that it's not very polite to use a parallel
> fetching tool under these circumstances...

I would bet that has more to do with traffic shaping on your connection 
to the external world than anything to do with the local bandwidth, in 
which case you could probably parallel all you want w/o improving 
download performance.


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Ralph Slooten]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> There is some examples in /etc/make.conf.example, but not for prozilla ...

If you are using the latest prozilla:

RESUMECOMMAND='/usr/local/bin/proz -r --no-curses --no-getch -s ${URI}
- -P ${DISTDIR} --min-size=2048'

This will invoke an ftp mirror search if the filesize if above 2MB (I do
this because the time taken for a < 2MB is less than the search for
faster mirrors). Adjust to your likings ;-)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFEA3KTCt0ZF9kLPvYRAgalAJsFip3T5V7XWRFNed2f0LH2rXUKCACgpBYV
hFsSjn1c/mHztaYOjDdidSw=
=A5u8
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Zac Slade]

On Monday 27 February 2006 10:39, El Nino wrote:
> > if you want to use prozilla, just change the  FETCHCOMMAND and
> > RESUMECOMMAND in /etc/make.conf
> >
> > There is some examples in /etc/make.conf.example, but not for prozilla
> how can i get to know all these portage features?
Read two lines above this one for the answer to your question.  The other way 
is to read man portage, man emerge, man make.conf.  The example file stays 
updated with new features (it may lag some) and so do the man pages.
-- 
Zac Slade
krakrjak@volumehost.net
ICQ:1415282 YM:krakrjak AIM:ttyp99
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Walter Dnes]

On Mon, Feb 27, 2006 at 05:05:39PM +0100, Alexander Skwar wrote
> El Nino wrote:
> 
> >   is there a way to accelerate the fetching part of emerge
> > by using prozilla or some other tool?
> 
> I never quite understood the sense in those tools.
> 
> Why should "prozilla or some other tool" make the
> download be faster? When I download something with
> wget, or watch emerge invoking wget, it's always
> maxing out the saturation of the line.

  I have a different interpretation.  Assume I'm doing an
emerge --deep --update --world

  I've set my ADSL router-modem to log off after 15 minutes.  The
sequence of events is something like...

  1) emerge wants to download a package, so it attempts to connect to a
     server, taking several seconds to wake up my ADSL router-modem

  2) emerge downloads a big package, taking a few minutes to do so

  3) emerge spends the next half hour building the big package, during
     which time the modem-router logs off

  4) emerge finally finishes building the package, and wants to work on
     the next one... GOTO 1

  I could run a short script

#!/bin/bash
emerge --deep --update --world --fetchonly
emerge --deep --update --world

...but I'd like to get an emerge going on the 1st package as soon as
it's finished downloading, whilst having the downloads of all the other
packages continue in a separate thread.  When the 1st build is finished,
check whether the 2nd package has been downloaded.  If not, wait.  Then
build the 2nd package... etc, etc.

  The best way to describe it is as a --fetchonly emerge that launches a
separate emerge as each individual package is finished downloading.  The
"build" emerges should be serialized, i.e. only one build running at a
time, because a package may depend on the immediately preceding package.

-- 
Walter Dnes <waltdnes@waltdnes.org> In linux /sbin/init is Job #1
My musings on technology and security at http://tech_sec.blog.ca
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Iain Buchanan]

On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> 
>   I have a different interpretation.

I assume you know about FEATURES="parallel-fetch"?
-- 
Iain Buchanan <iain at netspace dot net dot au>

"Elves and Dragons!" I says to him.  "Cabbages and potatoes are better
for you and me."
		-- J. R. R. Tolkien

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Mike Owen]

On 2/28/06, Iain Buchanan <iaindb@netspace.net.au> wrote:
> On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> >
> >   I have a different interpretation.
>
> I assume you know about FEATURES="parallel-fetch"?

It's probably not the best idea to recommend ~arch versions of portage.

Mike

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Boyd Stephen Smith Jr.]

On Tuesday 28 February 2006 16:27, "Walter Dnes" <waltdnes@waltdnes.org> 
wrote about 'Re: [gentoo-user] accelerate emerge':
> #!/bin/bash
> emerge --deep --update --world --fetchonly
> emerge --deep --update --world
>
> ...but I'd like to get an emerge going on the 1st package as soon as
> it's finished downloading, whilst having the downloads of all the other
> packages continue in a separate thread.

Try adding '&' to your first line to start it as a background process.  I 
did this for months.  MAKE SURE emerge --info SHOWS THAT YOU ARE USING THE 
DISTLOCKS FEATURE.

It's a separate process instead of a separate thread, but it should do what 
you want.  You may also want to throw a '> /dev/null 2>&-' before the '&', 
but that depends on your sensibilities.

-- 
"If there's one thing we've established over the years,
it's that the vast majority of our users don't have the slightest
clue what's best for them in terms of package stability."
-- Gentoo Developer Ciaran McCreesh
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Boyd Stephen Smith Jr.]

On Tuesday 28 February 2006 17:25, "Mike Owen" <kyphros@gmail.com> wrote 
about 'Re: [gentoo-user] accelerate emerge':
> On 2/28/06, Iain Buchanan <iaindb@netspace.net.au> wrote:
> > On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> > >   I have a different interpretation.
> >
> > I assume you know about FEATURES="parallel-fetch"?
>
> It's probably not the best idea to recommend ~arch versions of portage.

I will recommend going ~arch for portage, I've done so for over a year.

That said, there's no need to go ~arch just for this feature.  See my other 
post.

-- 
"If there's one thing we've established over the years,
it's that the vast majority of our users don't have the slightest
clue what's best for them in terms of package stability."
-- Gentoo Developer Ciaran McCreesh
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Iain Buchanan]

On Tue, 2006-02-28 at 15:25 -0800, Mike Owen wrote:
> On 2/28/06, Iain Buchanan <iaindb@netspace.net.au> wrote:
> > On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> > >
> > >   I have a different interpretation.
> >
> > I assume you know about FEATURES="parallel-fetch"?
> 
> It's probably not the best idea to recommend ~arch versions of portage.

Why not?  Its a great feature.  My (two) systems have been running
entirely ~x86 ever since I installed them (years ago), and I've never
had any real problems.  Only rarely do I have to recompile, or
downgrade, but usually it's all handled by portage anyway.

Standard disclaimer applies: just because it works now, doesn't mean it
will work in the future; not for production / isolated servers, etc.

Note that from the comments before you modify ~arch:

"... '~arch' is a superset of 'arch' which includes the unstable, in
testing, packages ... 'Broken' packages will not be added to testing ...
IF YOU ARE UNSURE OF YOUR ARCH, OR THE IMPLICATIONS, DO NOT MODIFY
[ACCEPT_KEYWORDS]".

:)
-- 
Iain Buchanan <iain at netspace dot net dot au>

The use of money is all the advantage there is to having money.
		-- B. Franklin

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Michael A. Smith]

El Nino wrote:
> AybOwan!
> 
>   is there a way to accelerate the fetching part of emerge
> by using prozilla or some other tool?

Someone else answered that question directly, so I'll try the 
indirect approach.

Another way to speed up emerges is to download while you compile:

emerge -fq foo & emerge foo

Of course it only makes sense if 'foo' has at least one dependency, 
but if you use this command frequently it makes sense to alias it.

Concurrent downloads used to be difficult, but now that Portage has 
locking it's reasonably safe. Still, every optimization comes with an 
increase in risk, and concurrent downloading is no different.

Peace,
Mike
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Michael A. Smith]

Iain Buchanan wrote:
> On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> 
>>  I have a different interpretation.
> 
> 
> I assume you know about FEATURES="parallel-fetch"?
How long has this been around? I couldn't find any mention of it in 
the man files. Where is it documented?

Thanks,
Mike
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] accelerate emerge

[Vladimir G. Ivanovic]

It's new in the latest testing version of emerge.

--- Vladimir

On Tue, 2006-02-28 at 22:17 -0500, Michael A. Smith wrote:
> Iain Buchanan wrote:
> > On Tue, 2006-02-28 at 17:27 -0500, Walter Dnes wrote:
> > 
> >>  I have a different interpretation.
> > 
> > 
> > I assume you know about FEATURES="parallel-fetch"?
> How long has this been around? I couldn't find any mention of it in 
> the man files. Where is it documented?
> 
> Thanks,
> Mike
-- 
Vladimir G. Ivanovic <vladimir@acm.org>

-- 
gentoo-user@gentoo.org mailing list