[gentoo-user] identical drives, different free space!

[gentoo-user] identical drives, different free space!

[Iain Buchanan]

Hi,

I have two 160Gb drives, one internal and one USB.  I've partitioned
them the same and created an identical filesystem on the USB drive for
backing up my internal drive.

I'm using the following rsync command to make the backup:
sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
--delete --delete-excluded --partial
--human-readable / /media/root-backup

however, after running this command sporadically for a few days, the USB
partition is now full, whereas my root partition isn't!

sda is internal, and sdd is external.  sda7 is the one I'm interested
in:

$ sudo fdisk -l

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000080

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          11       88326    6  FAT16
/dev/sda2   *          12        4875    39070080    b  W95 FAT32
/dev/sda3            4876        4888      104422+  83  Linux
/dev/sda4            4889       19457   117025492+   5  Extended
/dev/sda5            4889        7321    19543041   83  Linux
/dev/sda6            7322        7384      506016   83  Linux
/dev/sda7            7385       19457    96976341   83  Linux

Disk /dev/sdd: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5d5d0036

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1          11       88326    6  FAT16
/dev/sdd2              12        4875    39070080    b  W95 FAT32
/dev/sdd3            4876        4888      104422+  83  Linux
/dev/sdd4            4889       19457   117025492+   5  Extended
/dev/sdd5            4889        7321    19543041   83  Linux
/dev/sdd6            7322        7384      506016   83  Linux
/dev/sdd7            7385       19457    96976341   83  Linux

I just deleted a bunch of /var/tmp and distfiles to free up some space,
and ran the rsync again.  Now it looks like this:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                 92G   81G  6.1G  93% /
/dev/sdd7              92G   89G  4.6M 100% /media/root-backup

/dev/sda3              99M   39M   55M  42% /boot
/dev/sdd3              99M   39M   55M  42% /media/boot-backup

I'm doing the /root backup from cron, but the /boot backup manually when
I make changes.

I thought perhaps the ext3 options were different (ie. different amount
of "reserved" space) but that would make the "Avail" columns different,
and shouldn't make the "Used" columns different.

any thoughts as to why my USB partition is full?  thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Most people have two reasons for doing anything -- a good reason, and
the real reason.

Re: [gentoo-user] identical drives, different free space!

[Kaddeh]

[-- Attachment #1: Type: text/plain, Size: 3402 bytes --]

Are you doing a full recursive copy of / from rootfs for sdd7 (aka cp -r /)
if so, are the other partitions mounted as well?

Cheers

Kad

On Thu, May 13, 2010 at 6:51 PM, Iain Buchanan <iaindb@netspace.net.au>wrote:

> Hi,
>
> I have two 160Gb drives, one internal and one USB.  I've partitioned
> them the same and created an identical filesystem on the USB drive for
> backing up my internal drive.
>
> I'm using the following rsync command to make the backup:
> sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
> --delete --delete-excluded --partial
> --human-readable / /media/root-backup
>
> however, after running this command sporadically for a few days, the USB
> partition is now full, whereas my root partition isn't!
>
> sda is internal, and sdd is external.  sda7 is the one I'm interested
> in:
>
> $ sudo fdisk -l
>
> Disk /dev/sda: 160.0 GB, 160041885696 bytes
> 255 heads, 63 sectors/track, 19457 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000080
>
>   Device Boot      Start         End      Blocks   Id  System
> /dev/sda1               1          11       88326    6  FAT16
> /dev/sda2   *          12        4875    39070080    b  W95 FAT32
> /dev/sda3            4876        4888      104422+  83  Linux
> /dev/sda4            4889       19457   117025492+   5  Extended
> /dev/sda5            4889        7321    19543041   83  Linux
> /dev/sda6            7322        7384      506016   83  Linux
> /dev/sda7            7385       19457    96976341   83  Linux
>
> Disk /dev/sdd: 160.0 GB, 160041885696 bytes
> 255 heads, 63 sectors/track, 19457 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x5d5d0036
>
>   Device Boot      Start         End      Blocks   Id  System
> /dev/sdd1               1          11       88326    6  FAT16
> /dev/sdd2              12        4875    39070080    b  W95 FAT32
> /dev/sdd3            4876        4888      104422+  83  Linux
> /dev/sdd4            4889       19457   117025492+   5  Extended
> /dev/sdd5            4889        7321    19543041   83  Linux
> /dev/sdd6            7322        7384      506016   83  Linux
> /dev/sdd7            7385       19457    96976341   83  Linux
>
> I just deleted a bunch of /var/tmp and distfiles to free up some space,
> and ran the rsync again.  Now it looks like this:
>
> $ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> rootfs                 92G   81G  6.1G  93% /
> /dev/sdd7              92G   89G  4.6M 100% /media/root-backup
>
> /dev/sda3              99M   39M   55M  42% /boot
> /dev/sdd3              99M   39M   55M  42% /media/boot-backup
>
> I'm doing the /root backup from cron, but the /boot backup manually when
> I make changes.
>
> I thought perhaps the ext3 options were different (ie. different amount
> of "reserved" space) but that would make the "Avail" columns different,
> and shouldn't make the "Used" columns different.
>
> any thoughts as to why my USB partition is full?  thanks,
> --
> Iain Buchanan <iaindb at netspace dot net dot au>
>
> Most people have two reasons for doing anything -- a good reason, and
> the real reason.
>
>
>

[-- Attachment #2: Type: text/html, Size: 4015 bytes --]

Re: [gentoo-user] identical drives, different free space!

[Iain Buchanan]

On Thu, 2010-05-13 at 20:22 -0700, Kaddeh wrote:
> Are you doing a full recursive copy of / from rootfs for sdd7 (aka cp
> -r /) if so, are the other partitions mounted as well?

[snip]

yes, but the rsync command "-x" or "--one-file-system" should stop rsync
traversing to different mounts so (I hope) this should only copy the one
partition.

thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

When you have eliminated the impossible, whatever remains, however improbable,
must be the truth.
		-- Sherlock Holmes, "The Sign of Four"

Re: [gentoo-user] identical drives, different free space!

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

On Fri, 14 May 2010 11:21:02 +0930, Iain Buchanan wrote:

> I'm using the following rsync command to make the backup:
> sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
> --delete --delete-excluded --partial
> --human-readable / /media/root-backup

As the rsync command is failing with disk full, files are not being
deleted. Try adding --delete-before to the options to have old files
cleaned up before copying new ones.


-- 
Neil Bothwick

A consultant is a person who borrows your watch, tells you what time it
is, pockets the watch, and sends you a bill for it.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] identical drives, different free space!

[Iain Buchanan]

On Fri, 2010-05-14 at 09:35 +0100, Neil Bothwick wrote:
> On Fri, 14 May 2010 11:21:02 +0930, Iain Buchanan wrote:
> 
> > I'm using the following rsync command to make the backup:
> > sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
> > --delete --delete-excluded --partial
> > --human-readable / /media/root-backup
> 
> As the rsync command is failing with disk full, files are not being
> deleted. Try adding --delete-before to the options to have old files
> cleaned up before copying new ones.

that's what I thought initially, hence:

On Fri, 2010-05-14 at 11:21 +0930, Iain Buchanan wrote: 
> I just deleted a bunch of /var/tmp and distfiles to free up some space,
> and ran the rsync again.  Now it looks like this:
> 
> $ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> rootfs                 92G   81G  6.1G  93% /
> /dev/sdd7              92G   89G  4.6M 100% /media/root-backup
> 
> /dev/sda3              99M   39M   55M  42% /boot
> /dev/sdd3              99M   39M   55M  42% /media/boot-backup

So the last rsync didn't fail with "disk full" - it's got about 3G left
for use by root.

Any other ideas?  thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Yesterday I was a dog.  Today I'm a dog.  Tomorrow I'll probably still
be a dog. Sigh!  There's so little hope for advancement.
		-- Snoopy

Re: [gentoo-user] identical drives, different free space!

[scott n-h]

[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]

On Fri, May 14, 2010 at 7:11 PM, Iain Buchanan <iaindb@netspace.net.au>wrote:

> On Fri, 2010-05-14 at 09:35 +0100, Neil Bothwick wrote:
> > On Fri, 14 May 2010 11:21:02 +0930, Iain Buchanan wrote:
> >
> > > I'm using the following rsync command to make the backup:
> > > sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
> > > --delete --delete-excluded --partial
> > > --human-readable / /media/root-backup
> >
> > As the rsync command is failing with disk full, files are not being
> > deleted. Try adding --delete-before to the options to have old files
> > cleaned up before copying new ones.
>
> that's what I thought initially, hence:
>
> On Fri, 2010-05-14 at 11:21 +0930, Iain Buchanan wrote:
> > I just deleted a bunch of /var/tmp and distfiles to free up some space,
> > and ran the rsync again.  Now it looks like this:
> >
> > $ df -h
> > Filesystem            Size  Used Avail Use% Mounted on
> > rootfs                 92G   81G  6.1G  93% /
> > /dev/sdd7              92G   89G  4.6M 100% /media/root-backup
> >
> > /dev/sda3              99M   39M   55M  42% /boot
> > /dev/sdd3              99M   39M   55M  42% /media/boot-backup
>
> So the last rsync didn't fail with "disk full" - it's got about 3G left
> for use by root.
>
> Any other ideas?  thanks,
> --
> Iain Buchanan <iaindb at netspace dot net dot au>
>
> Yesterday I was a dog.  Today I'm a dog.  Tomorrow I'll probably still
> be a dog. Sigh!  There's so little hope for advancement.
>                -- Snoopy
>
>
> Have you checked to see if it is following symlinks? Possibly add a -l
option to "copy symlinks as symlinks"

[-- Attachment #2: Type: text/html, Size: 2200 bytes --]

Re: [gentoo-user] identical drives, different free space!

[Iain Buchanan]

On Sat, 2010-05-15 at 01:35 -0700, scott n-h wrote:

>         
> Have you checked to see if it is following symlinks? Possibly add a -l
> option to "copy symlinks as symlinks"

good idea, I didn't have the -l option.  Now I run rsync like this:

sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAlx --exclude suspend_file
--delete --delete-excluded --delete-before --partial --human-readable /
"${MOUNTPT}" >>"${LOGFILE}"

Note the "-l" AND "--delete-before".

However I'm STILL filling up the second drive for some unknown reason.

I've added "--exclude /usr/portage/distfiles" to the rsync options,
since there's no need to back up my distfiles, but I'd like to know why
it's not working...

-- 
Iain Buchanan <iain at pcorp dot com dot au>

It doesn't matter whether you win or lose -- until you lose.

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Iain Buchanan]

So after I excluded distfiles from my rsync, I found that the two
partitions had roughly the same free space... strange!  How could
excluding around 6G of distfiles make two copies of the same thing the
same size?

Well, it turns out I have the distfiles mounted with --bind to my
ftp/pub directory.  And looking in the rsync man page:

-x, --one-file-system
...
Also keep in mind that rsync treats a "bind" mount to
              the same device as being on the same filesystem.

So my distfiles were being copied in /usr/portage as well
as /home/ftp/pub!

Unfortunately the only way to get around it seems to be another
--exclude directive.  At least I understand what's going on now :)

thanks for all the suggestions,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Mr. Cole's Axiom:
	The sum of the intelligence on the planet is a constant; the
	population is growing.

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

On Mon, 17 May 2010 11:21:50 +0930, Iain Buchanan wrote:

> Well, it turns out I have the distfiles mounted with --bind to my
> ftp/pub directory.  And looking in the rsync man page:

Why not set $DISTDIR to the true location of distfiles instead of using
bind mounts?


-- 
Neil Bothwick

Tribble math: * + * = ***********************************

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[David W Noon]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

On Mon, 17 May 2010 10:10:02 +0200, Neil Bothwick wrote about Re:
[gentoo-user] [SOLVED] identical drives, different free space!:

>On Mon, 17 May 2010 11:21:50 +0930, Iain Buchanan wrote:
>
>> Well, it turns out I have the distfiles mounted with --bind to my
>> ftp/pub directory.  And looking in the rsync man page:
>
>Why not set $DISTDIR to the true location of distfiles instead of using
>bind mounts?

Because binding the directory to /home/ftp/pub makes the distfiles
available to the rest of one's network via anonymous ftp.  I do the
same thing here, without the "pub" subdirectory, and exclude /home/ftp/
from my backups.
-- 
Regards,

Dave  [RLU #314465]
======================================================================
dwnoon@ntlworld.com (David W Noon)
======================================================================

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Iain Buchanan]

On Mon, 2010-05-17 at 09:07 +0100, Neil Bothwick wrote:
> On Mon, 17 May 2010 11:21:50 +0930, Iain Buchanan wrote:
> 
> > Well, it turns out I have the distfiles mounted with --bind to my
> > ftp/pub directory.  And looking in the rsync man page:
> 
> Why not set $DISTDIR to the true location of distfiles instead of using
> bind mounts?

because /usr/portage/distfiles IS the real location,
and /home/ftp/pub/gentoo/distfiles is the ftp shared location.  vsftpd
doesn't handle symlinks, so I have to bind it.

Now that you mention it though, I could move it for real
into /home/ftp/pub/gentoo/distfiles and change DISTDIR... hm.

-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Real programmers don't bring brown-bag lunches.  If the vending machine
doesn't sell it, they don't eat it.  Vending machines don't sell quiche.

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

On Mon, 17 May 2010 12:31:17 +0100, David W Noon wrote:

> >> Well, it turns out I have the distfiles mounted with --bind to my
> >> ftp/pub directory.  And looking in the rsync man page:  
> >
> >Why not set $DISTDIR to the true location of distfiles instead of using
> >bind mounts?  
> 
> Because binding the directory to /home/ftp/pub makes the distfiles
> available to the rest of one's network via anonymous ftp.  I do the
> same thing here, without the "pub" subdirectory, and exclude /home/ftp/
> from my backups.

So the distfiles are actually in /usr/portage/distfiles?

I share my distfiles but I don't use FTP as that means storing copies of
the same file on each computer. Instead, I use NFS. /mnt/portage is
shared across all machines on the network and DISTDIR is set
to /mnt/portage/distfiles in each make.conf.

Sharing /mnt/portage like this means I can also share my overlay across
the network at /mnt/portage/local.


-- 
Neil Bothwick

Top Oxymorons Number 18: Taped live

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Iain Buchanan]

On Mon, 2010-05-17 at 12:39 +0100, Neil Bothwick wrote:
> On Mon, 17 May 2010 12:31:17 +0100, David W Noon wrote:

...

> So the distfiles are actually in /usr/portage/distfiles?

for me yes, it looks the same for David.

> I share my distfiles but I don't use FTP as that means storing copies of
> the same file on each computer. Instead, I use NFS. /mnt/portage is
> shared across all machines on the network and DISTDIR is set
> to /mnt/portage/distfiles in each make.conf.
> 
> Sharing /mnt/portage like this means I can also share my overlay across
> the network at /mnt/portage/local.

Until I pick up my laptop and drive to work, where network speeds to my
server drop from 100Mbit to 50kbit and I need that local copy!

Which is why I'm glad there are multiple ways to do it :)
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

   Old robot: I choose to believe what I was programmed to believe.

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 861 bytes --]

On Mon, 17 May 2010 21:50:28 +0930, Iain Buchanan wrote:

> > I share my distfiles but I don't use FTP as that means storing copies
> > of the same file on each computer. Instead, I use NFS. /mnt/portage is
> > shared across all machines on the network and DISTDIR is set
> > to /mnt/portage/distfiles in each make.conf.


> Until I pick up my laptop and drive to work, where network speeds to my
> server drop from 100Mbit to 50kbit and I need that local copy!

I tend not to run emerges when away from home, although the lack of a
local copy does prove awkward after a kernel upgrade that requires a
rebuild of the wireless drivers. Not a situation I have to deal with any
more, thankfully.

> Which is why I'm glad there are multiple ways to do it :)

Indeed :)


-- 
Neil Bothwick

A chicken is an egg's way of producing more eggs.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[David W Noon]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

On Mon, 17 May 2010 13:50:02 +0200, Neil Bothwick wrote about Re:
[gentoo-user] [SOLVED] identical drives, different free space!:

>On Mon, 17 May 2010 12:31:17 +0100, David W Noon wrote:
>
>> >> Well, it turns out I have the distfiles mounted with --bind to my
>> >> ftp/pub directory.  And looking in the rsync man page:  
>> >
>> >Why not set $DISTDIR to the true location of distfiles instead of
>> >using bind mounts?  
>> 
>> Because binding the directory to /home/ftp/pub makes the distfiles
>> available to the rest of one's network via anonymous ftp.  I do the
>> same thing here, without the "pub" subdirectory, and
>> exclude /home/ftp/ from my backups.
>
>So the distfiles are actually in /usr/portage/distfiles?

Correct.

>I share my distfiles but I don't use FTP as that means storing copies
>of the same file on each computer. Instead, I use NFS. /mnt/portage is
>shared across all machines on the network and DISTDIR is set
>to /mnt/portage/distfiles in each make.conf.

I used to do that, but it meant my NFS server had to be running to
perform any software maintenance on any box, so it became a single point
of failure. The FTP approach allows each box to be self-reliant.

>Sharing /mnt/portage like this means I can also share my overlay across
>the network at /mnt/portage/local.

My boxes have different stuff in their overlays, and one uses no
overlay packages at all.  Sharing overlays doesn't make much sense for
my set-up.
-- 
Regards,

Dave  [RLU #314465]
======================================================================
dwnoon@ntlworld.com (David W Noon)
======================================================================

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1324 bytes --]

On Mon, 17 May 2010 19:33:18 +0100, David W Noon wrote:

> >I share my distfiles but I don't use FTP as that means storing copies
> >of the same file on each computer. Instead, I use NFS. /mnt/portage is
> >shared across all machines on the network and DISTDIR is set
> >to /mnt/portage/distfiles in each make.conf.  
> 
> I used to do that, but it meant my NFS server had to be running to
> perform any software maintenance on any box, so it became a single point
> of failure. The FTP approach allows each box to be self-reliant.

Fair comment. I have DISTDIR on my mail server, so if that goes down,
I've more to worry about that a few tarballs. Even if it is inaccessible,
the other computers would simply download the files to the local
directory.

> >Sharing /mnt/portage like this means I can also share my overlay across
> >the network at /mnt/portage/local.  
> 
> My boxes have different stuff in their overlays, and one uses no
> overlay packages at all.  Sharing overlays doesn't make much sense for
> my set-up.

It makes sense for me because everything is in one place, making
maintenance and backups simpler. Even if a package is only used on one
computer, for now, a central location still makes sense.


-- 
Neil Bothwick

The computer revolution is over. The computers won.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Bill Kenworthy]

On Mon, 2010-05-17 at 21:53 +0100, Neil Bothwick wrote:
> On Mon, 17 May 2010 19:33:18 +0100, David W Noon wrote:
> 
> > >I share my distfiles but I don't use FTP as that means storing copies
> > >of the same file on each computer. Instead, I use NFS. /mnt/portage is
> > >shared across all machines on the network and DISTDIR is set
> > >to /mnt/portage/distfiles in each make.conf.  
> > 
> > I used to do that, but it meant my NFS server had to be running to
> > perform any software maintenance on any box, so it became a single point
> > of failure. The FTP approach allows each box to be self-reliant.
> 
> Fair comment. I have DISTDIR on my mail server, so if that goes down,
> I've more to worry about that a few tarballs. Even if it is inaccessible,
> the other computers would simply download the files to the local
> directory.
> 
> > >Sharing /mnt/portage like this means I can also share my overlay across
> > >the network at /mnt/portage/local.  
> > 
> > My boxes have different stuff in their overlays, and one uses no
> > overlay packages at all.  Sharing overlays doesn't make much sense for
> > my set-up.
> 
> It makes sense for me because everything is in one place, making
> maintenance and backups simpler. Even if a package is only used on one
> computer, for now, a central location still makes sense.
> 


As an alternative check out http-replicator - yes the clients do
download to a local directory but that can be cleaned afterwards.  It
also allows download locally when you know you are taking the machine
(laptop?) elsewhere.  An advantage over NFS is it seems to handle
parallel downloads of the same file so you can transparently build all
machines in parallel without the downloads stepping on each other over a
common NFS mount.

I also use a tmfs store for distfiles on one machine with plenty of ram
so thats a self-cleaning (on reboot :) alternative.

I have used NFS as well and its ok for data stores http-replicator is
much better.  Beware - NFS can be slow and flakey if used for building
over (/var/tmp/portage).

The great thing about gentoo's build system is its so flexible!

BillK

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Peter Humphrey]

On Tuesday 18 May 2010 04:13:07 Bill Kenworthy wrote:

> As an alternative check out http-replicator - yes the clients do
> download to a local directory but that can be cleaned afterwards.  It
> also allows download locally when you know you are taking the machine
> (laptop?) elsewhere.

Yet another approach is to have an rsync server on your LAN. In my case 
it's the local server box (print, squid, mail etc). It's simple to set 
up and to use, and all the boxes on the LAN can operate in their own 
devious ways.

-- 
Rgds
Peter.

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[William Kenworthy]

On Tue, 2010-05-18 at 10:12 +0100, Peter Humphrey wrote:
> On Tuesday 18 May 2010 04:13:07 Bill Kenworthy wrote:
> 
> > As an alternative check out http-replicator - yes the clients do
> > download to a local directory but that can be cleaned afterwards.  It
> > also allows download locally when you know you are taking the machine
> > (laptop?) elsewhere.
> 
> Yet another approach is to have an rsync server on your LAN. In my case 
> it's the local server box (print, squid, mail etc). It's simple to set 
> up and to use, and all the boxes on the LAN can operate in their own 
> devious ways.
> 

The advantage of http-replicator is that it is a caching proxy - if it
isnt in the cache, it downloads it and then serves it out to one or more
clients - rsync/FTP/wget/... can just share whats already there, not go
get the file in the first place.

BillK

-- 
William Kenworthy <billk@iinet.net.au>
Home in Perth!

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

On Tue, 18 May 2010 18:19:06 +0800, William Kenworthy wrote:

> The advantage of http-replicator is that it is a caching proxy - if it
> isnt in the cache, it downloads it and then serves it out to one or more
> clients - rsync/FTP/wget/... can just share whats already there, not go
> get the file in the first place.

What happens if the proxy is not available, such as when a laptop is away
from home? With NFS, the DISTDIR share simply isn't mounted and files are
downloaded to the local directory, there's no configuration switch needed
when away from home.

Caching the files locally does have its advantages, but for me the only
computer that would benefit from it, my netbook, is the one with the
least storage space to spare.


-- 
Neil Bothwick

I'm not opinionated, I'm just always right!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Peter Humphrey]

On Tuesday 18 May 2010 11:19:06 William Kenworthy wrote:

> The advantage of http-replicator is that it is a caching proxy - if
> it isnt in the cache, it downloads it and then serves it out to one
> or more clients - rsync/FTP/wget/... can just share whats already
> there, not go get the file in the first place.

My setup does exactly the same, since squid is running on the same box.

-- 
Rgds
Peter.

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[William Kenworthy]

On Tue, 2010-05-18 at 11:30 +0100, Peter Humphrey wrote:
> On Tuesday 18 May 2010 11:19:06 William Kenworthy wrote:
> 
> > The advantage of http-replicator is that it is a caching proxy - if
> > it isnt in the cache, it downloads it and then serves it out to one
> > or more clients - rsync/FTP/wget/... can just share whats already
> > there, not go get the file in the first place.
> 
> My setup does exactly the same, since squid is running on the same box.
> 

How have you configured it? - I wouldn't have though squid suitable
considering its designed for a different purpose and so regularly
expires items in its cache (i.e., they will be available for a limited
time before being cleaned.)  If you extend max_age, then it becomes
unsuitable as a regular web proxy/cache unless you are running multiple
instances.  There are posts saying that squid doesnt work well with
portage but other than a high miss rate (possibly because the files
expired?), no details are given.

Squid also seems to store its files named something
like /var/cache/squid/00/00/000000B9 so its hard to get at them directly
without having squid to serve them up while the http-replicator cache is
just the raw files - same as "distfiles" in fact.

see http://forums.gentoo.org/viewtopic.php?p=1138287#1138287 for details
on http-replicator.

BillK

-- 
William Kenworthy <billk@iinet.net.au>
Home in Perth!

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Peter Humphrey]

On Tuesday 18 May 2010 12:59:28 William Kenworthy wrote:

> On Tue, 2010-05-18 at 11:30 +0100, Peter Humphrey wrote:
> > My setup does exactly the same, since squid is running on the same
> > box.
> 
> How have you configured it? - I wouldn't have though squid suitable
> considering its designed for a different purpose and so regularly
> expires items in its cache (i.e., they will be available for a
> limited time before being cleaned.)  If you extend max_age, then it
> becomes unsuitable as a regular web proxy/cache unless you are
> running multiple instances.  There are posts saying that squid
> doesnt work well with portage but other than a high miss rate
> (possibly because the files expired?), no details are given.

In view of what you say, maybe I ought to look into http-replicator. I 
have noticed some quite large files being fetched when I thought they 
ought already to be in squid's cache.

-- 
Rgds
Peter.