[gentoo-user] identical drives, different free space!

[gentoo-user] identical drives, different free space!

[Iain Buchanan]

Hi,

I have two 160Gb drives, one internal and one USB.  I've partitioned
them the same and created an identical filesystem on the USB drive for
backing up my internal drive.

I'm using the following rsync command to make the backup:
sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
--delete --delete-excluded --partial
--human-readable / /media/root-backup

however, after running this command sporadically for a few days, the USB
partition is now full, whereas my root partition isn't!

sda is internal, and sdd is external.  sda7 is the one I'm interested
in:

$ sudo fdisk -l

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000080

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          11       88326    6  FAT16
/dev/sda2   *          12        4875    39070080    b  W95 FAT32
/dev/sda3            4876        4888      104422+  83  Linux
/dev/sda4            4889       19457   117025492+   5  Extended
/dev/sda5            4889        7321    19543041   83  Linux
/dev/sda6            7322        7384      506016   83  Linux
/dev/sda7            7385       19457    96976341   83  Linux

Disk /dev/sdd: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x5d5d0036

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1          11       88326    6  FAT16
/dev/sdd2              12        4875    39070080    b  W95 FAT32
/dev/sdd3            4876        4888      104422+  83  Linux
/dev/sdd4            4889       19457   117025492+   5  Extended
/dev/sdd5            4889        7321    19543041   83  Linux
/dev/sdd6            7322        7384      506016   83  Linux
/dev/sdd7            7385       19457    96976341   83  Linux

I just deleted a bunch of /var/tmp and distfiles to free up some space,
and ran the rsync again.  Now it looks like this:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                 92G   81G  6.1G  93% /
/dev/sdd7              92G   89G  4.6M 100% /media/root-backup

/dev/sda3              99M   39M   55M  42% /boot
/dev/sdd3              99M   39M   55M  42% /media/boot-backup

I'm doing the /root backup from cron, but the /boot backup manually when
I make changes.

I thought perhaps the ext3 options were different (ie. different amount
of "reserved" space) but that would make the "Avail" columns different,
and shouldn't make the "Used" columns different.

any thoughts as to why my USB partition is full?  thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Most people have two reasons for doing anything -- a good reason, and
the real reason.

Re: [gentoo-user] identical drives, different free space!

[Kaddeh]

[-- Attachment #1: Type: text/plain, Size: 3402 bytes --]

Are you doing a full recursive copy of / from rootfs for sdd7 (aka cp -r /)
if so, are the other partitions mounted as well?

Cheers

Kad

On Thu, May 13, 2010 at 6:51 PM, Iain Buchanan <iaindb@netspace.net.au>wrote:

> Hi,
>
> I have two 160Gb drives, one internal and one USB.  I've partitioned
> them the same and created an identical filesystem on the USB drive for
> backing up my internal drive.
>
> I'm using the following rsync command to make the backup:
> sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
> --delete --delete-excluded --partial
> --human-readable / /media/root-backup
>
> however, after running this command sporadically for a few days, the USB
> partition is now full, whereas my root partition isn't!
>
> sda is internal, and sdd is external.  sda7 is the one I'm interested
> in:
>
> $ sudo fdisk -l
>
> Disk /dev/sda: 160.0 GB, 160041885696 bytes
> 255 heads, 63 sectors/track, 19457 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x00000080
>
>   Device Boot      Start         End      Blocks   Id  System
> /dev/sda1               1          11       88326    6  FAT16
> /dev/sda2   *          12        4875    39070080    b  W95 FAT32
> /dev/sda3            4876        4888      104422+  83  Linux
> /dev/sda4            4889       19457   117025492+   5  Extended
> /dev/sda5            4889        7321    19543041   83  Linux
> /dev/sda6            7322        7384      506016   83  Linux
> /dev/sda7            7385       19457    96976341   83  Linux
>
> Disk /dev/sdd: 160.0 GB, 160041885696 bytes
> 255 heads, 63 sectors/track, 19457 cylinders
> Units = cylinders of 16065 * 512 = 8225280 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disk identifier: 0x5d5d0036
>
>   Device Boot      Start         End      Blocks   Id  System
> /dev/sdd1               1          11       88326    6  FAT16
> /dev/sdd2              12        4875    39070080    b  W95 FAT32
> /dev/sdd3            4876        4888      104422+  83  Linux
> /dev/sdd4            4889       19457   117025492+   5  Extended
> /dev/sdd5            4889        7321    19543041   83  Linux
> /dev/sdd6            7322        7384      506016   83  Linux
> /dev/sdd7            7385       19457    96976341   83  Linux
>
> I just deleted a bunch of /var/tmp and distfiles to free up some space,
> and ran the rsync again.  Now it looks like this:
>
> $ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> rootfs                 92G   81G  6.1G  93% /
> /dev/sdd7              92G   89G  4.6M 100% /media/root-backup
>
> /dev/sda3              99M   39M   55M  42% /boot
> /dev/sdd3              99M   39M   55M  42% /media/boot-backup
>
> I'm doing the /root backup from cron, but the /boot backup manually when
> I make changes.
>
> I thought perhaps the ext3 options were different (ie. different amount
> of "reserved" space) but that would make the "Avail" columns different,
> and shouldn't make the "Used" columns different.
>
> any thoughts as to why my USB partition is full?  thanks,
> --
> Iain Buchanan <iaindb at netspace dot net dot au>
>
> Most people have two reasons for doing anything -- a good reason, and
> the real reason.
>
>
>

[-- Attachment #2: Type: text/html, Size: 4015 bytes --]

Re: [gentoo-user] identical drives, different free space!

[Iain Buchanan]

On Thu, 2010-05-13 at 20:22 -0700, Kaddeh wrote:
> Are you doing a full recursive copy of / from rootfs for sdd7 (aka cp
> -r /) if so, are the other partitions mounted as well?

[snip]

yes, but the rsync command "-x" or "--one-file-system" should stop rsync
traversing to different mounts so (I hope) this should only copy the one
partition.

thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

When you have eliminated the impossible, whatever remains, however improbable,
must be the truth.
		-- Sherlock Holmes, "The Sign of Four"

Re: [gentoo-user] identical drives, different free space!

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 604 bytes --]

On Fri, 14 May 2010 11:21:02 +0930, Iain Buchanan wrote:

> I'm using the following rsync command to make the backup:
> sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
> --delete --delete-excluded --partial
> --human-readable / /media/root-backup

As the rsync command is failing with disk full, files are not being
deleted. Try adding --delete-before to the options to have old files
cleaned up before copying new ones.


-- 
Neil Bothwick

A consultant is a person who borrows your watch, tells you what time it
is, pockets the watch, and sends you a bill for it.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] identical drives, different free space!

[Iain Buchanan]

On Fri, 2010-05-14 at 09:35 +0100, Neil Bothwick wrote:
> On Fri, 14 May 2010 11:21:02 +0930, Iain Buchanan wrote:
> 
> > I'm using the following rsync command to make the backup:
> > sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
> > --delete --delete-excluded --partial
> > --human-readable / /media/root-backup
> 
> As the rsync command is failing with disk full, files are not being
> deleted. Try adding --delete-before to the options to have old files
> cleaned up before copying new ones.

that's what I thought initially, hence:

On Fri, 2010-05-14 at 11:21 +0930, Iain Buchanan wrote: 
> I just deleted a bunch of /var/tmp and distfiles to free up some space,
> and ran the rsync again.  Now it looks like this:
> 
> $ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> rootfs                 92G   81G  6.1G  93% /
> /dev/sdd7              92G   89G  4.6M 100% /media/root-backup
> 
> /dev/sda3              99M   39M   55M  42% /boot
> /dev/sdd3              99M   39M   55M  42% /media/boot-backup

So the last rsync didn't fail with "disk full" - it's got about 3G left
for use by root.

Any other ideas?  thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Yesterday I was a dog.  Today I'm a dog.  Tomorrow I'll probably still
be a dog. Sigh!  There's so little hope for advancement.
		-- Snoopy

Re: [gentoo-user] identical drives, different free space!

[scott n-h]

[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]

On Fri, May 14, 2010 at 7:11 PM, Iain Buchanan <iaindb@netspace.net.au>wrote:

> On Fri, 2010-05-14 at 09:35 +0100, Neil Bothwick wrote:
> > On Fri, 14 May 2010 11:21:02 +0930, Iain Buchanan wrote:
> >
> > > I'm using the following rsync command to make the backup:
> > > sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file
> > > --delete --delete-excluded --partial
> > > --human-readable / /media/root-backup
> >
> > As the rsync command is failing with disk full, files are not being
> > deleted. Try adding --delete-before to the options to have old files
> > cleaned up before copying new ones.
>
> that's what I thought initially, hence:
>
> On Fri, 2010-05-14 at 11:21 +0930, Iain Buchanan wrote:
> > I just deleted a bunch of /var/tmp and distfiles to free up some space,
> > and ran the rsync again.  Now it looks like this:
> >
> > $ df -h
> > Filesystem            Size  Used Avail Use% Mounted on
> > rootfs                 92G   81G  6.1G  93% /
> > /dev/sdd7              92G   89G  4.6M 100% /media/root-backup
> >
> > /dev/sda3              99M   39M   55M  42% /boot
> > /dev/sdd3              99M   39M   55M  42% /media/boot-backup
>
> So the last rsync didn't fail with "disk full" - it's got about 3G left
> for use by root.
>
> Any other ideas?  thanks,
> --
> Iain Buchanan <iaindb at netspace dot net dot au>
>
> Yesterday I was a dog.  Today I'm a dog.  Tomorrow I'll probably still
> be a dog. Sigh!  There's so little hope for advancement.
>                -- Snoopy
>
>
> Have you checked to see if it is following symlinks? Possibly add a -l
option to "copy symlinks as symlinks"

[-- Attachment #2: Type: text/html, Size: 2200 bytes --]

Re: [gentoo-user] identical drives, different free space!

[Iain Buchanan]

On Sat, 2010-05-15 at 01:35 -0700, scott n-h wrote:

>         
> Have you checked to see if it is following symlinks? Possibly add a -l
> option to "copy symlinks as symlinks"

good idea, I didn't have the -l option.  Now I run rsync like this:

sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAlx --exclude suspend_file
--delete --delete-excluded --delete-before --partial --human-readable /
"${MOUNTPT}" >>"${LOGFILE}"

Note the "-l" AND "--delete-before".

However I'm STILL filling up the second drive for some unknown reason.

I've added "--exclude /usr/portage/distfiles" to the rsync options,
since there's no need to back up my distfiles, but I'd like to know why
it's not working...

-- 
Iain Buchanan <iain at pcorp dot com dot au>

It doesn't matter whether you win or lose -- until you lose.

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Iain Buchanan]

So after I excluded distfiles from my rsync, I found that the two
partitions had roughly the same free space... strange!  How could
excluding around 6G of distfiles make two copies of the same thing the
same size?

Well, it turns out I have the distfiles mounted with --bind to my
ftp/pub directory.  And looking in the rsync man page:

-x, --one-file-system
...
Also keep in mind that rsync treats a "bind" mount to
              the same device as being on the same filesystem.

So my distfiles were being copied in /usr/portage as well
as /home/ftp/pub!

Unfortunately the only way to get around it seems to be another
--exclude directive.  At least I understand what's going on now :)

thanks for all the suggestions,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Mr. Cole's Axiom:
	The sum of the intelligence on the planet is a constant; the
	population is growing.

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

On Mon, 17 May 2010 11:21:50 +0930, Iain Buchanan wrote:

> Well, it turns out I have the distfiles mounted with --bind to my
> ftp/pub directory.  And looking in the rsync man page:

Why not set $DISTDIR to the true location of distfiles instead of using
bind mounts?


-- 
Neil Bothwick

Tribble math: * + * = ***********************************

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] [SOLVED] identical drives, different free space!

[Iain Buchanan]

On Mon, 2010-05-17 at 09:07 +0100, Neil Bothwick wrote:
> On Mon, 17 May 2010 11:21:50 +0930, Iain Buchanan wrote:
> 
> > Well, it turns out I have the distfiles mounted with --bind to my
> > ftp/pub directory.  And looking in the rsync man page:
> 
> Why not set $DISTDIR to the true location of distfiles instead of using
> bind mounts?

because /usr/portage/distfiles IS the real location,
and /home/ftp/pub/gentoo/distfiles is the ftp shared location.  vsftpd
doesn't handle symlinks, so I have to bind it.

Now that you mention it though, I could move it for real
into /home/ftp/pub/gentoo/distfiles and change DISTDIR... hm.

-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Real programmers don't bring brown-bag lunches.  If the vending machine
doesn't sell it, they don't eat it.  Vending machines don't sell quiche.