[gentoo-dev] FW: rsync etiquette follow-up

[gentoo-dev] FW: rsync etiquette follow-up

[Stuart Herbert]

Dear all,

I originally sent the comments below to GWN as feedback on the recent call
for rsync etiquette.  Kurt Lieber [klieber@gentoo.org] asked me to forward
on to this list, for further discussion.

Please be gentle - this is my first posting to the list ;-)

Best regards,
Stu
--

-----Original Message-----
From: Stuart Herbert [mailto:stuart@gnqs.org] 
Sent: 09 May 2003 11:24
To: 'gwn-feedback@gentoo.org'
Cc: 'stuart@gnqs.org'
Subject: rsync etiquette follow-up


Hi there,

I read with great interest your article on the need for rsync etiquette.
I'm sure lots of people have already written to you making the same point.
I'd like to add my voice to this point.

Gentoo's 'emerge' is a wonderful tool in many ways - but when it comes to a
site containing multiple machines, improvements in its design could help
reduce the load on your rsync servers.

I run four Gentoo boxes at the moment, and they're all connected to the
Internet through a single firewall configured with masquerading.  If I were
to rsync each machine just once a day, I wonder whether it would look like
one machine had rsync'd four times from the point of view of the Gentoo
rsync mirror that I use?

Anyway, whenever I install a new package onto a machine, it is important to
my work that *each* of these machines has the same version of the package.
The machines are different x86 architectures, so building a binary package
on the one machine isn't my preferred choice.  The whole point of using
Gentoo is that each machine runs code that is specifically optimised for the
hardware.

To achieve this, there's no getting away from it.  I have to 'emerge rsync'
on each machine, and 'emerge <package>'.  The 'emerge rsync' is there to
ensure that each machine picks up the same version of both the package, and
its dependencies.  This can be done via cron once every couple of days.  But
it still means that I end up hitting your rsync mirrors *once* for every
Gentoo machine I run.  I also end up hitting your distfile mirrors *once*
per machine as well.

I know that I could run an rsync mirror just for internal use - and that
would help a lot.  Running a distfiles mirror is a lot less practical.  It
would be much better if there was a way to share '/usr/portage' across
multiple machines.  You can't do this safely via NFS.  If two machines try
to download the same distfile at the same time, they interfere with each
other.  

It'd be much better if emerge could be changed to a client/server model,
where the emerge command becomes a client that contacts the (possibly
remote) emerge server to do all the downloading and rsyncing.  For people
running both client/server on the same machine, there's no noticable
difference.  For people like me, trying to run a site of machines, I'm able
to reduce the amount of load I place on your rsync servers and your distfile
servers.

This gives me another advantage.  I don't have to have a (potentially large)
/usr/portage/distfiles on each machine.  If three of my machines are
clients, and the server only runs on the fourth machine, I can setup a cron
job to clear out /usr/portage/distfiles on each of the clients on a nightly
basis - and keep my diskpage usage down.  (As an aside, it'd be great to see
/usr/portage moved into /var.  One of my few true disappointments with
Gentoo is having to have /usr mounted read-write a lot of the time)

I hope that I've explained it right - and that some intrepid emerge
developer will take on the task.  And soon.

Best regards,
Stu
--



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] FW: rsync etiquette follow-up

[Steven Lucy]

On Mon, May 12, 2003 at 04:56:37PM +0100, Stuart Herbert <stuart@myrddraal.demon.co.uk> said
> I know that I could run an rsync mirror just for internal use - and that
> would help a lot.  Running a distfiles mirror is a lot less practical.  It
> would be much better if there was a way to share '/usr/portage' across
> multiple machines.  You can't do this safely via NFS.  If two machines try
> to download the same distfile at the same time, they interfere with each
> other.  
> [...]
> (As an aside, it'd be great to see
> /usr/portage moved into /var.  One of my few true disappointments with
> Gentoo is having to have /usr mounted read-write a lot of the time)

you can set DISTDIR in /etc/make.conf to point to whatever you want.  mine
points to /var/distfiles.  also, if you shared /usr/portage (and
/var/distfiles, but not /var/db, etc.) via NFS and always emerged new
packages on one machine first (so only it would do the fetching), then
initiated the emerges on the other machine they would find the tarball and
skip right to the unpack/compile stages.  that's what i would do.

Steven

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] FW: rsync etiquette follow-up

[Priit Laes]

On Mon, 12 May 2003 16:56:37 +0100
"Stuart Herbert" <stuart@myrddraal.demon.co.uk> wrote:

> Dear all,
> 
> I originally sent the comments below to GWN as feedback on the recent call
> for rsync etiquette.  Kurt Lieber [klieber@gentoo.org] asked me to forward
> on to this list, for further discussion.
> 
> Please be gentle - this is my first posting to the list ;-)
> 
> Best regards,
> Stu
> --
> 
> -----Original Message-----
> From: Stuart Herbert [mailto:stuart@gnqs.org] 
> Sent: 09 May 2003 11:24
> To: 'gwn-feedback@gentoo.org'
> Cc: 'stuart@gnqs.org'
> Subject: rsync etiquette follow-up
> 
> 
> Hi there,
> 
> I read with great interest your article on the need for rsync etiquette.
> I'm sure lots of people have already written to you making the same point.
> I'd like to add my voice to this point.
> 
> Gentoo's 'emerge' is a wonderful tool in many ways - but when it comes to a
> site containing multiple machines, improvements in its design could help
> reduce the load on your rsync servers.
> 
> I run four Gentoo boxes at the moment, and they're all connected to the
> Internet through a single firewall configured with masquerading.  If I were
> to rsync each machine just once a day, I wonder whether it would look like
> one machine had rsync'd four times from the point of view of the Gentoo
> rsync mirror that I use?
> 
> Anyway, whenever I install a new package onto a machine, it is important to
> my work that *each* of these machines has the same version of the package.
> The machines are different x86 architectures, so building a binary package
> on the one machine isn't my preferred choice.  The whole point of using
> Gentoo is that each machine runs code that is specifically optimised for the
> hardware.
Portage directory (/usr/portage) contains only new packages...
/var/db/pkg contains packages installed installed on machine...

-- 
Priit Laes <amdATttDOTee>                  _o)
http://amd-core.tk                         /\\  _o)  _o)
Why don't they fix bugs in bugzilla???    _\_V _( ) _( )

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] FW: rsync etiquette follow-up

[Sven Vermeulen]

[-- Attachment #1: Type: text/plain, Size: 1472 bytes --]

On Mon, May 12, 2003 at 04:56:37PM +0100, Stuart Herbert wrote:
> I know that I could run an rsync mirror just for internal use - and that
> would help a lot.  

I also recommend it from the moment you have 2 boxes, since it will provide
you with the most optimal synchronous Portage stage on both boxes.

> Running a distfiles mirror is a lot less practical.  It
> would be much better if there was a way to share '/usr/portage' across
> multiple machines.  You can't do this safely via NFS.  If two machines try
> to download the same distfile at the same time, they interfere with each
> other.  

There are several ways. I personally have /usr/portage/distfiles NFS-exported
and _read-only_. I have also set FETCH_COMMAND in /etc/make.conf to something
similar to the following:

FETCHCOMMAND="/usr/bin/ssh -l portage niandra /usr/bin/wget -O
		\${DISTDIR}/\${FILE} \${URI}"

This will log in into niandra (my testbox) and use wget to download the
source and place it into niandra:/usr/portage/distfiles, which is NFS
exported.

I say "similar" since I'm actually using a C-program, but the above should
work.

Wkr,
	Sven Vermeulen

-- 
Thanks to DRM, you know that something has been built in environment of 
unspecified degree of security, from source you cannot check, written by 
programmers you don't know, released after passing QA of unknown quality and 
which is released under a license that disclaims any responsibility...

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] FW: rsync etiquette follow-up

[Robin H.Johnson]

[-- Attachment #1: Type: text/plain, Size: 2088 bytes --]

On Mon, May 12, 2003 at 04:56:37PM +0100, Stuart Herbert wrote:
> I know that I could run an rsync mirror just for internal use - and that
> would help a lot.  Running a distfiles mirror is a lot less practical.  It
> would be much better if there was a way to share '/usr/portage' across
> multiple machines.  You can't do this safely via NFS.  If two machines try
> to download the same distfile at the same time, they interfere with each
> other.  
For the moment, could we have a workaround?
1. portage should keep lockfiles while it makes a call to wget/$FETCH so
that duplicate instances don't blow up (you can cause this even on a
single machine, no need for NFS).
2. as something extra from the above, say you are merging a bunch of
packages, and one of them is being downloaded by an alternate process.
Instead of failing immediately, portage should find the NEXT thing that
doesn't have an dependancies in the depgraph (another pure leaf node so
to speak), and run that way. If there is nothing else that can be done
without waiting for the other existing process, then we can throw a
temporary error.

> It'd be much better if emerge could be changed to a client/server model,
> where the emerge command becomes a client that contacts the (possibly
> remote) emerge server to do all the downloading and rsyncing.  For people
> running both client/server on the same machine, there's no noticable
> difference.  For people like me, trying to run a site of machines, I'm able
> to reduce the amount of load I place on your rsync servers and your distfile
> servers.
Long term this would be ideal, esp. if merged with functionality like
redhat's RHN subscriptions (able to push updates to a group of machines
automatically, and get error messages if it fails). This is one of those
enterprise level things that we need.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]