[gentoo-user] rsync + tar + bz2 ?

[gentoo-user] rsync + tar + bz2 ?

[Grant]

I'm backing up numerous large files on another machine on my local
network.  I've only been using rsync, but it occured to me that I
might be able to save some time and space if I incorporate tar and
bzip2.  How will rsync interact with those?  If I turn the whole
backup into a big tar.bz2, would rsync need to redownload the whole
thing if I change one file?  If so, maybe I should turn different
groups of files into tar.bz2 archives so rsync only needs to
redownload an archive if one of its files has changed?

- Grant

Re: [gentoo-user] rsync + tar + bz2 ?

[Mike Kazantsev]

[-- Attachment #1: Type: text/plain, Size: 1710 bytes --]

On Sat, 7 Mar 2009 08:04:17 -0800
Grant <emailgrant@gmail.com> wrote:

> I'm backing up numerous large files on another machine on my local
> network.  I've only been using rsync, but it occured to me that I
> might be able to save some time and space if I incorporate tar and
> bzip2.  How will rsync interact with those?  If I turn the whole
> backup into a big tar.bz2, would rsync need to redownload the whole
> thing if I change one file?  If so, maybe I should turn different
> groups of files into tar.bz2 archives so rsync only needs to
> redownload an archive if one of its files has changed?

It's not a default behavior, but there is an '--inplace' option.

Also,there is a '--compress' option, if the bandwith is the only
problem, otherwise you can use lzma (with normal-best ratio) to either
acheive much better compression than bzip2 or still slightly better
ratio with improved speed / less cpu time with 'lzma -1' (fast mode).

And if you're going to put a lot of files (like whole fs) into a
single tar just to transfer it to some remote destination, prehaps you
shouldn't be using rsync at all, since you'll end up reading all the
files anyway to create the tar.
Alternatively, you can save disk space on the source machine by piping
tar directly to destination, with compression either on the source to
lessen the banwidth, or on the remote to lessen the load on the source
machine cpu.

That said, you can also use tar to create (or pipe) incremental backups
- just the changes since the time last one was made. Tar can handle that
as easily as rsync does, since it checks what needs to be transferred
each time anyway.

-- 
Mike Kazantsev // fraggod.net

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] rsync + tar + bz2 ?

[Grant]

>> I'm backing up numerous large files on another machine on my local
>> network.  I've only been using rsync, but it occured to me that I
>> might be able to save some time and space if I incorporate tar and
>> bzip2.  How will rsync interact with those?  If I turn the whole
>> backup into a big tar.bz2, would rsync need to redownload the whole
>> thing if I change one file?  If so, maybe I should turn different
>> groups of files into tar.bz2 archives so rsync only needs to
>> redownload an archive if one of its files has changed?
>
> It's not a default behavior, but there is an '--inplace' option.
>
> Also,there is a '--compress' option, if the bandwith is the only
> problem, otherwise you can use lzma (with normal-best ratio) to either
> acheive much better compression than bzip2 or still slightly better
> ratio with improved speed / less cpu time with 'lzma -1' (fast mode).
>
> And if you're going to put a lot of files (like whole fs) into a
> single tar just to transfer it to some remote destination, prehaps you
> shouldn't be using rsync at all, since you'll end up reading all the
> files anyway to create the tar.
> Alternatively, you can save disk space on the source machine by piping
> tar directly to destination, with compression either on the source to
> lessen the banwidth, or on the remote to lessen the load on the source
> machine cpu.
>
> That said, you can also use tar to create (or pipe) incremental backups
> - just the changes since the time last one was made. Tar can handle that
> as easily as rsync does, since it checks what needs to be transferred
> each time anyway.
>
> --
> Mike Kazantsev // fraggod.net

Good stuff, thanks a lot Mike.

- Grant

Re: [gentoo-user] rsync + tar + bz2 ?

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 673 bytes --]

Am Samstag, 7. März 2009 17:04:17 schrieb Grant:
> I'm backing up numerous large files on another machine on my local
> network.  I've only been using rsync, but it occured to me that I
> might be able to save some time and space if I incorporate tar and
> bzip2.  How will rsync interact with those?  If I turn the whole
> backup into a big tar.bz2, would rsync need to redownload the whole
> thing if I change one file?  If so, maybe I should turn different
> groups of files into tar.bz2 archives so rsync only needs to
> redownload an archive if one of its files has changed?

By using either rsh or ssh, tar can backup over the net, too.

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] rsync + tar + bz2 ?

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

Am Samstag, 7. März 2009 21:13:49 schrieb Dirk Heinrichs:
> Am Samstag, 7. März 2009 17:04:17 schrieb Grant:
> > I'm backing up numerous large files on another machine on my local
> > network.  I've only been using rsync, but it occured to me that I
> > might be able to save some time and space if I incorporate tar and
> > bzip2.  How will rsync interact with those?  If I turn the whole
> > backup into a big tar.bz2, would rsync need to redownload the whole
> > thing if I change one file?  If so, maybe I should turn different
> > groups of files into tar.bz2 archives so rsync only needs to
> > redownload an archive if one of its files has changed?
>
> By using either rsh or ssh, tar can backup over the net, too.

OTOH, I think rsync is still the better solution, because even for large files, 
it only sends the deltas. But in the end, you will be doing some time 
measurements to find the best solution, anyway ;-)

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] rsync + tar + bz2 ?

[Alejandro]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

2009/3/7 Dirk Heinrichs <dirk.heinrichs@online.de>

> Am Samstag, 7. März 2009 21:13:49 schrieb Dirk Heinrichs:
> > Am Samstag, 7. März 2009 17:04:17 schrieb Grant:
> > > I'm backing up numerous large files on another machine on my local
> > > network.  I've only been using rsync, but it occured to me that I
> > > might be able to save some time and space if I incorporate tar and
> > > bzip2.  How will rsync interact with those?  If I turn the whole
> > > backup into a big tar.bz2, would rsync need to redownload the whole
> > > thing if I change one file?  If so, maybe I should turn different
> > > groups of files into tar.bz2 archives so rsync only needs to
> > > redownload an archive if one of its files has changed?
> >
> > By using either rsh or ssh, tar can backup over the net, too.
>
> OTOH, I think rsync is still the better solution, because even for large
> files,
> it only sends the deltas. But in the end, you will be doing some time
> measurements to find the best solution, anyway ;-)
>
> Bye...
>
>        Dirk
>

rsync -z

        -z, --compress              compress file data during the transfer
            --compress-level=NUM    explicitly set compression level
            --skip-compress=LIST    skip compressing files with suffix in
LIST

[-- Attachment #2: Type: text/html, Size: 1756 bytes --]

Re: [gentoo-user] rsync + tar + bz2 ?

[Daniel Troeder]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

Am Samstag, den 07.03.2009, 08:04 -0800 schrieb Grant:
> I'm backing up numerous large files on another machine on my local
> network.  I've only been using rsync, but it occured to me that I
> might be able to save some time and space if I incorporate tar and
> bzip2.  How will rsync interact with those?  If I turn the whole
> backup into a big tar.bz2, would rsync need to redownload the whole
> thing if I change one file?  If so, maybe I should turn different
> groups of files into tar.bz2 archives so rsync only needs to
> redownload an archive if one of its files has changed?
> 
> - Grant

You may be interested in duplicity: http://duplicity.nongnu.org/

Bye,
Daniel


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] rsync + tar + bz2 ?

[Dirk Heinrichs]

[-- Attachment #1: Type: text/plain, Size: 822 bytes --]

Am Samstag, 7. März 2009 17:04:17 schrieb Grant:
> I'm backing up numerous large files on another machine on my local
> network.  I've only been using rsync, but it occured to me that I
> might be able to save some time and space if I incorporate tar and
> bzip2.  How will rsync interact with those?  If I turn the whole
> backup into a big tar.bz2, would rsync need to redownload the whole
> thing if I change one file?  If so, maybe I should turn different
> groups of files into tar.bz2 archives so rsync only needs to
> redownload an archive if one of its files has changed?

Another way, although a bit more work to setup, whould be to use a "Network 
block device". Unlike NFS, the server just exports the block device, 
everything else (mkfs, encryption) can be done on the client.

Bye...

	Dirk

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-user] rsync + tar + bz2 ?

[Hung Dang]

Dirk Heinrichs wrote:
> Am Samstag, 7. März 2009 17:04:17 schrieb Grant:
>   
>> I'm backing up numerous large files on another machine on my local
>> network.  I've only been using rsync, but it occured to me that I
>> might be able to save some time and space if I incorporate tar and
>> bzip2.  How will rsync interact with those?  If I turn the whole
>> backup into a big tar.bz2, would rsync need to redownload the whole
>> thing if I change one file?  If so, maybe I should turn different
>> groups of files into tar.bz2 archives so rsync only needs to
>> redownload an archive if one of its files has changed?
>>     
>
> Another way, although a bit more work to setup, whould be to use a "Network 
> block device". Unlike NFS, the server just exports the block device, 
> everything else (mkfs, encryption) can be done on the client.
>
> Bye...
>
> 	Dirk
>   
rsync will download only if source and destination files are different.
 From my experiences using rar is faster and save more space  than bz2.

Hung

Re: [gentoo-user] rsync + tar + bz2 ?

[Saphirus Sage]

Hung Dang wrote:
> Dirk Heinrichs wrote:
>> Am Samstag, 7. März 2009 17:04:17 schrieb Grant:
>>  
>>> I'm backing up numerous large files on another machine on my local
>>> network.  I've only been using rsync, but it occured to me that I
>>> might be able to save some time and space if I incorporate tar and
>>> bzip2.  How will rsync interact with those?  If I turn the whole
>>> backup into a big tar.bz2, would rsync need to redownload the whole
>>> thing if I change one file?  If so, maybe I should turn different
>>> groups of files into tar.bz2 archives so rsync only needs to
>>> redownload an archive if one of its files has changed?
>>>     
>>
>> Another way, although a bit more work to setup, whould be to use a
>> "Network block device". Unlike NFS, the server just exports the block
>> device, everything else (mkfs, encryption) can be done on the client.
>>
>> Bye...
>>
>>     Dirk
>>   
> rsync will download only if source and destination files are different.
> From my experiences using rar is faster and save more space  than bz2.
>
> Hung
>
But rar is a proprietary archival format, I'd much sooner go with a tar,
compressed with bzip2 or lzma. If the biggest concern is just getting it
done quickly, gzip it, but for the love of all things free, not rar, I say!

Re: [gentoo-user] rsync + tar + bz2 ?

[Enrico Weigelt]

* Grant <emailgrant@gmail.com> wrote:

> I'm backing up numerous large files on another machine on my 
> local network. 

Let me suggest another idea: 

Put a venti on the remote machine and push the tree to be backed
up with vac. Never think about maintaining incremental backups
anymore - just pull in the stuff, and venti will store equal 
data blocks only once. All you now have to do is to put the 
root score (printed out by vac) into some safe place. 
You can even directly mount the backup via vacfs :)


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------