Re: [gentoo-portage-dev] Feature request

Re: [gentoo-portage-dev] Feature request

[Marius Mauch]

On Wed, Dec 31, 2003 at 09:34:57AM -0500, Allen Parker wrote:
> I think it'd be a good idea for portage-ng to include better localization
> features (specifically relating to mirrors). I've recently been plagued with
> RESTRICT="nomirror" being set in recent mysql 4.0.x ebuilds causing my
> filefetching operations to take considerable amounts of time when they
> should actually be the fastest. (my example is 30kb/s from argentina instead
> of the regular 4Mb/s from gentoo.mirrors.pair.com)

portage-2.0.50 already has support several new files in /etc/portage, including a file to set local mirrors to override the nomirror restriction. There is no documentation for them yet though, search bugs.gentoo.org for "thirdpartymirrors" for a detailed explanation.

Marius

--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Feature request

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 999 bytes --]

On Wednesday 31 December 2003 15:34, Allen Parker wrote:
> I think it'd be a good idea for portage-ng to include better localization
> features (specifically relating to mirrors). I've recently been plagued
> with RESTRICT="nomirror" being set in recent mysql 4.0.x ebuilds causing my
> filefetching operations to take considerable amounts of time when they
> should actually be the fastest. (my example is 30kb/s from argentina
> instead of the regular 4Mb/s from gentoo.mirrors.pair.com)

To me this has more to do with the mirror selection algorithm than with 
localization. Mirrors from the mirror:// pseudo-protocol should be chosen in 
such a way that they are both load-balanced, and optimized for local users. 
Preferably this should work without user interference.

The same should hold for multiple source locations in the ebuild itself, but 
it is less important.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Feature request

[Scott Taylor]

Whatever came of comment #14 of bug 28806 - That is something along the
lines of what i'd like to see. In my own network, I have a central
server with its /usr/portage/distfiles available locally by anonymous
ftp. This way, my local mirror has any file used on itself (or
prefetched for my other workstations by way of an emerge --fetchonly).
So, I do have files on this machine that should be checked first,
regardless of whether a build is marked "nomirror". For any network that
has local repositories either ftp/nfs/samba/whatever could find that
useful too. Lists of other groups of mirrors (sourceforge, etc) are also
good things to allow customization on. Its just my luck to want some
huge source that it tries to get from a slow sourceforge mirror on the
other side of the world...


On Thu, 2004-01-01 at 12:54, Paul de Vrieze wrote:
> On Wednesday 31 December 2003 15:34, Allen Parker wrote:
> > I think it'd be a good idea for portage-ng to include better localization
> > features (specifically relating to mirrors). I've recently been plagued
> > with RESTRICT="nomirror" being set in recent mysql 4.0.x ebuilds causing my
> > filefetching operations to take considerable amounts of time when they
> > should actually be the fastest. (my example is 30kb/s from argentina
> > instead of the regular 4Mb/s from gentoo.mirrors.pair.com)
> 
> To me this has more to do with the mirror selection algorithm than with 
> localization. Mirrors from the mirror:// pseudo-protocol should be chosen in 
> such a way that they are both load-balanced, and optimized for local users. 
> Preferably this should work without user interference.
> 
> The same should hold for multiple source locations in the ebuild itself, but 
> it is less important.
> 
> Paul
--
Scott Taylor - <security@303underground.com> 

work, n.:
	The blessed respite from screaming kids and
	soap operas for which you actually get paid.

    


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Feature request

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1739 bytes --]

On Thursday 01 January 2004 21:14, Scott Taylor wrote:
> Whatever came of comment #14 of bug 28806 - That is something along the
> lines of what i'd like to see. In my own network, I have a central
> server with its /usr/portage/distfiles available locally by anonymous
> ftp. This way, my local mirror has any file used on itself (or
> prefetched for my other workstations by way of an emerge --fetchonly).
> So, I do have files on this machine that should be checked first,
> regardless of whether a build is marked "nomirror". For any network that
> has local repositories either ftp/nfs/samba/whatever could find that
> useful too. Lists of other groups of mirrors (sourceforge, etc) are also
> good things to allow customization on. Its just my luck to want some
> huge source that it tries to get from a slow sourceforge mirror on the
> other side of the world...

A stopgap that you could use is to install squid on the server node. Use it as 
a emerge only proxy. However you MUST make some config changes. Change the 
used real memory to a low value as it will not have any point for portage 
(you'll not download things again and again), but REALLY REALLY IMPORTANT
is to set the 

maximum size of cached items

to something that is in the 100 or even 200 mb range. (Standard is 1 mb or 
something like that, quite pointless for portage). If you give your caching 
proxy enough cache space it will be quite effective, and it even saves you 
maintaining the distfiles dir. It also stops the need for different schemes 
for allowing distfile requests from any host to get onto the master cache.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Feature request

[Scott Taylor]

I've even got a dedicated cache appliance for http, but files that had
been requested through ftp aren't cached there. I think probably the
first stopgap that i'd apply would be to comment out the line that
parses the "nomirror" option. If its used to keep files from being
pushed onto the rotation servers, great... but its an annoyance within
my network. Then i'd see about making my own server the first in the
list of sourceforge mirrors, followed by a mirror that i know works
better for me. Perhaps the mirrorselect tool could automatically check
for good sf.net servers too. For that matter, I'd quite like to see a
fill-in-the-blank line plus the ability to choose the order from within
mirrorselect... perhaps even a order of preference that round-robins
equally weighted servers. 

On Thu, 2004-01-01 at 13:24, Paul de Vrieze wrote:

> A stopgap that you could use is to install squid on the server node. Use it as 
> a emerge only proxy. However you MUST make some config changes. Change the 
> used real memory to a low value as it will not have any point for portage 
> (you'll not download things again and again), but REALLY REALLY IMPORTANT
> is to set the 
> 
> maximum size of cached items
> 
> to something that is in the 100 or even 200 mb range. (Standard is 1 mb or 
> something like that, quite pointless for portage). If you give your caching 
> proxy enough cache space it will be quite effective, and it even saves you 
> maintaining the distfiles dir. It also stops the need for different schemes 
> for allowing distfile requests from any host to get onto the master cache.
> 
> Paul
--
Scott Taylor - <security@303underground.com> 

BOFH Excuse #448:

vi needs to be upgraded to vii

    


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Feature request

[Paul de Vrieze]

[-- Attachment #1: signed data --]
[-- Type: text/plain, Size: 1415 bytes --]

On Thursday 01 January 2004 21:43, Scott Taylor wrote:
> I've even got a dedicated cache appliance for http, but files that had
> been requested through ftp aren't cached there. I think probably the
> first stopgap that i'd apply would be to comment out the line that
> parses the "nomirror" option. If its used to keep files from being
> pushed onto the rotation servers, great... but its an annoyance within
> my network. Then i'd see about making my own server the first in the
> list of sourceforge mirrors, followed by a mirror that i know works
> better for me. Perhaps the mirrorselect tool could automatically check
> for good sf.net servers too. For that matter, I'd quite like to see a
> fill-in-the-blank line plus the ability to choose the order from within
> mirrorselect... perhaps even a order of preference that round-robins
> equally weighted servers.

Squid can cache ftp too (and others), given that you set the filesize to a 
size that is reasonable for portage (and not really for webbrowsing, by 
default squid doesn't cache most source files).

This does not mean that mirror selection can and should be done better, 
however it is hard to come up with a real good algorithm that does not 
involve people changing things by themselves (but still allows them to)

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Feature request

[Scott Taylor]

The default list of sourceforge (and other) mirrors would seem to me to
be well suited to being in make.globals where their value and syntax are
easily found, and new values can easily be placed in make.conf for those
who prefer other mirror sites. and bad mirrors can get yanked. Like for
example, every once in a while a gnukorea (or something like that) comes
up as an attempted mirror, however an emerge gets stuck there because
its asking for a password. If the lists of available mirrors were more
easily found and updated, it would be a lot easier to fix things like
that.

And I wouldn't expect it to be too difficult to add a
FEATURE="ignorenomirror". The machines inside my network only try my
local mirror, plus one of the gentoo mirrors before trying the
non-mirrored source uri anyway, so ignoring the nomirror directive is
not going to cause a drastic increase in file not found requests from
the mirrors. The server hosting the distfiles here would be left with a
normal set of source mirrors, and obeying the nomirror directive like
everyone else. 

I might be the only one trying to do it just like this, but I'm
certainly not the only one trying to share out my own distfiles
directory on an internal network. I've even considered burning distfiles
snapshots on dvd-r to leave in otherwise unused drives. Also, I'd heard
a request from catalyst that portage be able to understand directory
paths as a mirror source. Thats another fine example of using a local
distfile store of some sort that you would much prefer to have it try
first, while ignoring a nomirror directive.


On Thu, 2004-01-01 at 14:19, Paul de Vrieze wrote:
> This does not mean that mirror selection can and should be done better, 
> however it is hard to come up with a real good algorithm that does not 
> involve people changing things by themselves (but still allows them to)
> 
> Paul
--
Scott Taylor - <security@303underground.com> 

The Golden Rule of Arts and Sciences:
	He who has the gold makes the rules.

    


--
gentoo-portage-dev@gentoo.org mailing list

Re: [gentoo-portage-dev] Feature request

[Brian]

Have you looked at the tsp-cache -streaming distfile cache for lan use? 
Forum link
http://forums.gentoo.org/viewtopic.php?t=110973&highlight=download+server

It looks like it is working well for him.  It does however rely on the
tsp-cache server to be first in the /etc/make.conf mirrors list.  An
advantage is if it does not have it cached it will begin the download
and stream it to any users downstream simultaneously.

On Thu, 2004-01-01 at 12:14, Scott Taylor wrote:
> Whatever came of comment #14 of bug 28806 - That is something along the
> lines of what i'd like to see. In my own network, I have a central
> server with its /usr/portage/distfiles available locally by anonymous
> ftp. This way, my local mirror has any file used on itself (or
> prefetched for my other workstations by way of an emerge --fetchonly).
> So, I do have files on this machine that should be checked first,
> regardless of whether a build is marked "nomirror". For any network that
> has local repositories either ftp/nfs/samba/whatever could find that
> useful too. Lists of other groups of mirrors (sourceforge, etc) are also
> good things to allow customization on. Its just my luck to want some
> huge source that it tries to get from a slow sourceforge mirror on the
> other side of the world...
> 
> 
> On Thu, 2004-01-01 at 12:54, Paul de Vrieze wrote:
> > On Wednesday 31 December 2003 15:34, Allen Parker wrote:
> > > I think it'd be a good idea for portage-ng to include better localization
> > > features (specifically relating to mirrors). I've recently been plagued
> > > with RESTRICT="nomirror" being set in recent mysql 4.0.x ebuilds causing my
> > > filefetching operations to take considerable amounts of time when they
> > > should actually be the fastest. (my example is 30kb/s from argentina
> > > instead of the regular 4Mb/s from gentoo.mirrors.pair.com)
> > 
> > To me this has more to do with the mirror selection algorithm than with 
> > localization. Mirrors from the mirror:// pseudo-protocol should be chosen in 
> > such a way that they are both load-balanced, and optimized for local users. 
> > Preferably this should work without user interference.
> > 
> > The same should hold for multiple source locations in the ebuild itself, but 
> > it is less important.
> > 
> > Paul
> --
> Scott Taylor - <security@303underground.com> 
> 
> work, n.:
> 	The blessed respite from screaming kids and
> 	soap operas for which you actually get paid.
> 
>     
> 
> 
> --
> gentoo-portage-dev@gentoo.org mailing list
-- 
Brian <dol-sen@telus.net>


--
gentoo-portage-dev@gentoo.org mailing list

[gentoo-portage-dev] Feature request

[Allen Parker]

I think it'd be a good idea for portage-ng to include better localization
features (specifically relating to mirrors). I've recently been plagued with
RESTRICT="nomirror" being set in recent mysql 4.0.x ebuilds causing my
filefetching operations to take considerable amounts of time when they
should actually be the fastest. (my example is 30kb/s from argentina instead
of the regular 4Mb/s from gentoo.mirrors.pair.com)

Allen Dale Parker
allenp@efn.org

"ego sum ens ompnipotens" 



--
gentoo-portage-dev@gentoo.org mailing list