[gentoo-user] rsync to a USB stick

[gentoo-user] rsync to a USB stick

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 654 bytes --]

Hi All,

I run:

 rsync -a -l --delete -v /mnt/Business_dir /media/sdf1

to back up a directory from a PC to a USB stick.  However, from a cursory look 
this *seems* to copy the complete directory (every time I run it) and 
overwrites the USB stick.  Carrying on like this it will life-expire the USB 
stick in no time, plus it takes ages to complete as it copies over every 
single file again and again.

Is there a cleverer option I can add to rsync so that it only copies new 
files, overwrites older versions of the same and only deletes any files or 
directories that have been deleted from the source directory?
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] rsync to a USB stick

[meino.cramer]

Mick <michaelkintzios@gmail.com> [10-05-29 11:12]:
> Hi All,
> 
> I run:
> 
>  rsync -a -l --delete -v /mnt/Business_dir /media/sdf1
> 
> to back up a directory from a PC to a USB stick.  However, from a cursory look 
> this *seems* to copy the complete directory (every time I run it) and 
> overwrites the USB stick.  Carrying on like this it will life-expire the USB 
> stick in no time, plus it takes ages to complete as it copies over every 
> single file again and again.
> 
> Is there a cleverer option I can add to rsync so that it only copies new 
> files, overwrites older versions of the same and only deletes any files or 
> directories that have been deleted from the source directory?
> -- 
> Regards,
> Mick

Hi,

this is really a shot in the dark, but...
Would it possible, that the USB-stick is carrying a filesystem, which
lacks the support of enough ijnformations, rsync needs to operate
corrrectly (especially: file times?
Concerning the wear out of usb-sticks (and flash media in general):
There are specialised filesystems out there (dont remember what
exactly their names were), which take care of a "flashy" environment
(spread writes to different parts each time),

HTH!

Best regards,
mcc



-- 
Please don't send me any Word- or Powerpoint-Attachments
unless it's absolutely neccessary. - Send simply Text.
See http://www.gnu.org/philosophy/no-word-attachments.html
In a world without fences and walls nobody needs gates and windows.

Re: [gentoo-user] rsync to a USB stick

[Alan McKinnon]

On Saturday 29 May 2010 11:01:39 Mick wrote:
> Hi All,
> 
> I run:
> 
>  rsync -a -l --delete -v /mnt/Business_dir /media/sdf1
> 
> to back up a directory from a PC to a USB stick.  However, from a cursory
> look this *seems* to copy the complete directory (every time I run it) and
> overwrites the USB stick.  Carrying on like this it will life-expire the
> USB stick in no time, plus it takes ages to complete as it copies over
> every single file again and again.
> 
> Is there a cleverer option I can add to rsync so that it only copies new
> files, overwrites older versions of the same and only deletes any files or
> directories that have been deleted from the source directory?

What filesystem is on the disk?

If it's FAT, rsync will not benefit as FAT does not have any notion of the 
metadata that is on the PCs disk


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] rsync to a USB stick

[Alan McKinnon]

On Saturday 29 May 2010 11:01:39 Mick wrote:
> Hi All,
> 
> I run:
> 
>  rsync -a -l --delete -v /mnt/Business_dir /media/sdf1
> 
> to back up a directory from a PC to a USB stick.  However, from a cursory
> look this *seems* to copy the complete directory (every time I run it) and
> overwrites the USB stick.  Carrying on like this it will life-expire the
> USB stick in no time, plus it takes ages to complete as it copies over
> every single file again and again.
> 
> Is there a cleverer option I can add to rsync so that it only copies new
> files, overwrites older versions of the same and only deletes any files or
> directories that have been deleted from the source directory?


Arg, typo in previous post. I meant what filesystem is on the USB stick?


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] rsync to a USB stick

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1200 bytes --]

On Saturday 29 May 2010 10:30:54 Alan McKinnon wrote:
> On Saturday 29 May 2010 11:01:39 Mick wrote:
> > Hi All,
> >
> > I run:
> >
> >  rsync -a -l --delete -v /mnt/Business_dir /media/sdf1
> >
> > to back up a directory from a PC to a USB stick.  However, from a cursory
> > look this *seems* to copy the complete directory (every time I run it)
> > and overwrites the USB stick.  Carrying on like this it will life-expire
> > the USB stick in no time, plus it takes ages to complete as it copies
> > over every single file again and again.
> >
> > Is there a cleverer option I can add to rsync so that it only copies new
> > files, overwrites older versions of the same and only deletes any files
> > or directories that have been deleted from the source directory?
> 
> Arg, typo in previous post. I meant what filesystem is on the USB stick?

FAT32

Disk /dev/sdb: 8019 MB, 8019509248 bytes
20 heads, 16 sectors/track, 48947 cylinders
Units = cylinders of 320 * 512 = 163840 bytes
Disk identifier: 0xc3072e18

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1       48948     7831512    c  W95 FAT32 (LBA)
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] rsync to a USB stick

[Alan McKinnon]

On Saturday 29 May 2010 11:39:19 Mick wrote:
> On Saturday 29 May 2010 10:30:54 Alan McKinnon wrote:
> > On Saturday 29 May 2010 11:01:39 Mick wrote:
> > > Hi All,
> > > 
> > > I run:
> > >  rsync -a -l --delete -v /mnt/Business_dir /media/sdf1
> > > 
> > > to back up a directory from a PC to a USB stick.  However, from a
> > > cursory look this *seems* to copy the complete directory (every time I
> > > run it) and overwrites the USB stick.  Carrying on like this it will
> > > life-expire the USB stick in no time, plus it takes ages to complete
> > > as it copies over every single file again and again.
> > > 
> > > Is there a cleverer option I can add to rsync so that it only copies
> > > new files, overwrites older versions of the same and only deletes any
> > > files or directories that have been deleted from the source directory?
> > 
> > Arg, typo in previous post. I meant what filesystem is on the USB stick?
> 
> FAT32
> 
> Disk /dev/sdb: 8019 MB, 8019509248 bytes
> 20 heads, 16 sectors/track, 48947 cylinders
> Units = cylinders of 320 * 512 = 163840 bytes
> Disk identifier: 0xc3072e18
> 
>    Device Boot      Start         End      Blocks   Id  System
> /dev/sdb1   *           1       48948     7831512    c  W95 FAT32 (LBA)

Can you reformat to the same filesystem as the source disk and see if that 
makes a difference?


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] rsync to a USB stick

[Steven]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

On Sat, May 29, 2010 at 2:01 AM, Mick <michaelkintzios@gmail.com> wrote:

> Hi All,
>
> I run:
>
>  rsync -a -l --delete -v /mnt/Business_dir /media/sdf1
>
> to back up a directory from a PC to a USB stick.  However, from a cursory
> look
> this *seems* to copy the complete directory (every time I run it) and
> overwrites the USB stick.  Carrying on like this it will life-expire the
> USB
> stick in no time, plus it takes ages to complete as it copies over every
> single file again and again.
>
> Is there a cleverer option I can add to rsync so that it only copies new
> files, overwrites older versions of the same and only deletes any files or
> directories that have been deleted from the source directory?
> --
> Regards,
> Mick
>
Short answer "man rsync"
You'll find everything you need.
It is possible to sync files incrementally with rsync I just can't remember
how right now
Sorry really tired right now. Im sure someone will come a long with a more
appropriate answer.

[-- Attachment #2: Type: text/html, Size: 1361 bytes --]

Re: [gentoo-user] rsync to a USB stick

[Joshua Murphy]

On Sat, May 29, 2010 at 5:39 AM, Steven <apartment415@gmail.com> wrote:
>
>
> On Sat, May 29, 2010 at 2:01 AM, Mick <michaelkintzios@gmail.com> wrote:
>>
>> Hi All,
>>
>> I run:
>>
>>  rsync -a -l --delete -v /mnt/Business_dir /media/sdf1
>>
>> to back up a directory from a PC to a USB stick.  However, from a cursory
>> look
>> this *seems* to copy the complete directory (every time I run it) and
>> overwrites the USB stick.  Carrying on like this it will life-expire the
>> USB
>> stick in no time, plus it takes ages to complete as it copies over every
>> single file again and again.
>>
>> Is there a cleverer option I can add to rsync so that it only copies new
>> files, overwrites older versions of the same and only deletes any files or
>> directories that have been deleted from the source directory?
>> --
>> Regards,
>> Mick
>
> Short answer "man rsync"
> You'll find everything you need.
> It is possible to sync files incrementally with rsync I just can't remember
> how right now
> Sorry really tired right now. Im sure someone will come a long with a more
> appropriate answer.
>

Notably...
--checksum
--recursive

And *not* using (due to the limitations of FAT and FAT32)...
--archive (implies several others)
--perms
--times
--group
--owner
--whole-file

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] rsync to a USB stick

[Peter Ruskin]

On Saturday 29 May 2010 10:01:39 Mick wrote:
> Hi All,
>
> I run:
>
>  rsync -a -l --delete -v /mnt/Business_dir /media/sdf1
>
> to back up a directory from a PC to a USB stick.  However, from a
> cursory look this *seems* to copy the complete directory (every
> time I run it) and overwrites the USB stick.  Carrying on like
> this it will life-expire the USB stick in no time, plus it takes
> ages to complete as it copies over every single file again and
> again.
>
> Is there a cleverer option I can add to rsync so that it only
> copies new files, overwrites older versions of the same and only
> deletes any files or directories that have been deleted from the
> source directory?

Try adding a stroke after the directories:
  rsync -a -l --delete -v /mnt/Business_dir/ /media/sdf1/

Does that help?
-- 
Peter
========================================================================
Gentoo Linux: Portage 2.2_rc67			kernel-2.6.34-gentoo
AMD Phenom(tm) 9950 Quad-Core Processor		gcc(Gentoo: 4.4.3-r2)
KDE: 3.5.10					Qt: 3.3.8b
========================================================================

[gentoo-user] Re: rsync to a USB stick

[Remy Blank]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

Mick wrote:
> Is there a cleverer option I can add to rsync so that it only copies new 
> files, overwrites older versions of the same and only deletes any files or 
> directories that have been deleted from the source directory?

See the --modify-window option in the rsync man page. In particular, the
resolution of timestamps on FAT is 2 seconds, so you may want to use
--modify-window=1.

-- Remy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: rsync to a USB stick

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 559 bytes --]

On Saturday 29 May 2010 11:34:25 Remy Blank wrote:
> Mick wrote:
> > Is there a cleverer option I can add to rsync so that it only copies new
> > files, overwrites older versions of the same and only deletes any files
> > or directories that have been deleted from the source directory?
> 
> See the --modify-window option in the rsync man page. In particular, the
> resolution of timestamps on FAT is 2 seconds, so you may want to use
> --modify-window=1.

Nice one!  Will try this next time and see if it makes a difference.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] rsync to a USB stick

[David Relson]

On Sat, 29 May 2010 10:01:39 +0100
Mick wrote:

> Hi All,
> 
> I run:
> 
>  rsync -a -l --delete -v /mnt/Business_dir /media/sdf1
> 
> to back up a directory from a PC to a USB stick.  However, from a
> cursory look this *seems* to copy the complete directory (every time
> I run it) and overwrites the USB stick.  Carrying on like this it
> will life-expire the USB stick in no time, plus it takes ages to
> complete as it copies over every single file again and again.
> 
> Is there a cleverer option I can add to rsync so that it only copies
> new files, overwrites older versions of the same and only deletes any
> files or directories that have been deleted from the source directory?
> -- 
> Regards,
> Mick


For years I've used "rsync -Cavzu ..." to do updates.  That's been my
mantra for so long I don't recall what each option does do know that it
updates (rather than copies everything).

Indeed flash drives _do_ have a lifetime.  My recollection is that it's
in the thousands of writes if not the hundreds of thousands of writes.
Assuming a life of 1,000 writes and you backup once daily, that's 3
years of backups.  10,000 writes would be 30 years.  Of course if you
backup every hour, 10,000 writes is a year (or so).

Honestly, I've stopped worrying about manual copies to flash drives.  

Of course if you have a program that writes to a flash drive
frequently, that's a very different story ...

HTH,

David

Re: [gentoo-user] rsync to a USB stick

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

On Sat, 29 May 2010 07:59:31 -0400, David Relson wrote:

> Indeed flash drives _do_ have a lifetime.  My recollection is that it's
> in the thousands of writes if not the hundreds of thousands of writes.
> Assuming a life of 1,000 writes and you backup once daily, that's 3
> years of backups.  10,000 writes would be 30 years.  Of course if you
> backup every hour, 10,000 writes is a year (or so).

You're assuming that each backup only writes once, which is far from
true. If you mount a drive with the sync option, the FAT is updated for
every block you write, so even a single file can cause thousands of
writes to the same location.


-- 
Neil Bothwick

Electrocution, n.:
Burning at the stake with all the modern improvements.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] rsync to a USB stick

[David Relson]

On Sun, 30 May 2010 11:48:21 +0100
Neil Bothwick wrote:

> On Sat, 29 May 2010 07:59:31 -0400, David Relson wrote:
> 
> > Indeed flash drives _do_ have a lifetime.  My recollection is that
> > it's in the thousands of writes if not the hundreds of thousands of
> > writes. Assuming a life of 1,000 writes and you backup once daily,
> > that's 3 years of backups.  10,000 writes would be 30 years.  Of
> > course if you backup every hour, 10,000 writes is a year (or so).
> 
> You're assuming that each backup only writes once, which is far from
> true. If you mount a drive with the sync option, the FAT is updated
> for every block you write, so even a single file can cause thousands
> of writes to the same location.
> 
> 
> -- 
> Neil Bothwick

Neil,

Correct -- if the USB is mounted synchronously.  

Normally Linux uses asynchronous writes (caching), so will hit the FAT
much less often. 

I've tried synchronous writes and it's a real performance killer.

However for a DOS formatted stick (which is the norm) the FAT does seem
to be the week link.

David

[gentoo-user] Re: rsync to a USB stick

[Grant Edwards]

On 2010-05-30, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Sat, 29 May 2010 07:59:31 -0400, David Relson wrote:
>
>> Indeed flash drives _do_ have a lifetime.  My recollection is that it's
>> in the thousands of writes if not the hundreds of thousands of writes.
>> Assuming a life of 1,000 writes and you backup once daily, that's 3
>> years of backups.  10,000 writes would be 30 years.  Of course if you
>> backup every hour, 10,000 writes is a year (or so).
>
> You're assuming that each backup only writes once, which is far from
> true. If you mount a drive with the sync option, the FAT is updated for
> every block you write, so even a single file can cause thousands of
> writes to the same location.

And you're assuming that the flash controller chip in the USB drive
doesn't do wear-leavelling.

-- 
Grant

Re: [gentoo-user] Re: rsync to a USB stick

[David Relson]

On Sun, 30 May 2010 14:20:36 +0000 (UTC)
Grant Edwards wrote:

> On 2010-05-30, Neil Bothwick <neil@digimed.co.uk> wrote:
> > On Sat, 29 May 2010 07:59:31 -0400, David Relson wrote:
> >
> >> Indeed flash drives _do_ have a lifetime.  My recollection is that
> >> it's in the thousands of writes if not the hundreds of thousands
> >> of writes. Assuming a life of 1,000 writes and you backup once
> >> daily, that's 3 years of backups.  10,000 writes would be 30
> >> years.  Of course if you backup every hour, 10,000 writes is a
> >> year (or so).
> >
> > You're assuming that each backup only writes once, which is far from
> > true. If you mount a drive with the sync option, the FAT is updated
> > for every block you write, so even a single file can cause
> > thousands of writes to the same location.
> 
> And you're assuming that the flash controller chip in the USB drive
> doesn't do wear-leavelling.

FWIW, I have enabled synchronous writes for a Disk-On-Module (SSD)
formatted ext2. It makes writing take significantly longer and I have
had a DOM go bad (become unusable).  Admittedly, I don't know whether
the DOM does wear-levelling and I don't know the underlying cause of
the failure.  In any case it was "Not Good (tm)" ...

Re: [gentoo-user] Re: rsync to a USB stick

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

On Sun, 30 May 2010 14:20:36 +0000 (UTC), Grant Edwards wrote:

> > You're assuming that each backup only writes once, which is far from
> > true. If you mount a drive with the sync option, the FAT is updated
> > for every block you write, so even a single file can cause thousands
> > of writes to the same location.  
> 
> And you're assuming that the flash controller chip in the USB drive
> doesn't do wear-leavelling.

Even with wear levelling, writing in sync mode still does thousands of
writes. They may be more spread out, but there are still a lot more than
one per day and the previous assumptions are still false.


-- 
Neil Bothwick

Law of Mechanical Repair: After your hands become coated with
grease, your nose will begin to itch.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: rsync to a USB stick

[Grant Edwards]

On 2010-05-31, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Sun, 30 May 2010 14:20:36 +0000 (UTC), Grant Edwards wrote:
>
>> > You're assuming that each backup only writes once, which is far from
>> > true. If you mount a drive with the sync option, the FAT is updated
>> > for every block you write, so even a single file can cause thousands
>> > of writes to the same location.  
>> 
>> And you're assuming that the flash controller chip in the USB drive
>> doesn't do wear-leavelling.
>
> Even with wear levelling, writing in sync mode still does thousands of
> writes. They may be more spread out, but there are still a lot more than
> one per day and the previous assumptions are still false.

Agreed.  Sync writes will definitely wear out flash sooner, but it's
not as bad as one might think since flash controller chips _generally_
do wear levelling and may even do bad-block management that will swap
in spare blocks when wornw blocks start to go bad.  Of couse, none of
the USB thumb-drive vendors will ever spec any of that, so you have no
way of actually knowing.

-- 
Grant