[gentoo-user] File replication

[gentoo-user] File replication

[Stephen Liu]

Hi folks,

I'm prepared using an USB enclosure for data stroage.

I'm searching a tool/method whenever I save a file on /home/user a
duplicate copy will be saved automatically on
/mnt/point/of/usb_enclosure/home/user simultaneously, similar to
replication/mirroring.   Thus the USB enclosure can be used as a mobile
storage device and can be plugged in on any Linux PC.

Please advise.  TIA

B.R.
SL
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Michael Crute]

On 8/25/06, Stephen Liu <satimis@yahoo.com> wrote:
> Hi folks,
>
> I'm prepared using an USB enclosure for data stroage.
>
> I'm searching a tool/method whenever I save a file on /home/user a
> duplicate copy will be saved automatically on
> /mnt/point/of/usb_enclosure/home/user simultaneously, similar to
> replication/mirroring.   Thus the USB enclosure can be used as a mobile
> storage device and can be plugged in on any Linux PC.

My first thought for this kind of setup would be a little Perl magic
with inotify... not sure if something already exists like this.

http://www.edoceo.com/creo/inotify/perl-inotify.php

-Mike

-- 
________________________________
Michael E. Crute
http://mike.crute.org

I may not have gone where I intended to go, but I think I have ended
up where I intended to be. --Douglas Adams
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

On Sat, 26 Aug 2006 08:49:23 +0800 (CST), Stephen Liu wrote:

> I'm searching a tool/method whenever I save a file on /home/user a
> duplicate copy will be saved automatically on
> /mnt/point/of/usb_enclosure/home/user simultaneously, similar to
> replication/mirroring.   Thus the USB enclosure can be used as a mobile
> storage device and can be plugged in on any Linux PC.

You could set up a RAID-1 array on the two devices. Auto-detect RAID
information is saved at the end of the device, so it can still be used as
a standalone drive/partition when connected to something else. One
advantage of RAID is that if you run with the device disconnected and
then connect it, the RAID will be updated in the background.

I've not tried this, so don't blame me if you lose data or your cat
catches fire :(


-- 
Neil Bothwick

BUFFERS=20 FILES=15 2nd down, 4th quarter, 5 yards to go!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] File replication

[Robert Welz]

Neil Bothwick wrote:
> On Sat, 26 Aug 2006 08:49:23 +0800 (CST), Stephen Liu wrote:
> 
>> I'm searching a tool/method whenever I save a file on /home/user a
>> duplicate copy will be saved automatically on
>> /mnt/point/of/usb_enclosure/home/user simultaneously, similar to
>> replication/mirroring.   Thus the USB enclosure can be used as a mobile
>> storage device and can be plugged in on any Linux PC.
> 
> You could set up a RAID-1 array on the two devices. Auto-detect RAID
> information is saved at the end of the device, so it can still be used as
> a standalone drive/partition when connected to something else. One
> advantage of RAID is that if you run with the device disconnected and
> then connect it, the RAID will be updated in the background.

Yes, but syncing takes a lot time and should be observeved not to 
interrupt while in progress. You will need a script to notify you that 
syncing is done othewise you have a corrupt device.

What about a perl script that just copies all files in /home/user to the 
device once it is plugged in?

Robert
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Robert,

Tks for your advice.

> What about a perl script that just copies all files in /home/user to
> the 
> device once it is plugged in?

I'm not a programmer nor very acknowledgeable on perl.  Could you
please shed me some guide how to start.  What I need is on saving a new
file or edited file on /home/user a duplicate copy would be saved
simultaneously on the USB enclosure to the pre-selected directory.  TIA

B.R.
SL

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Bryan,

Tks for your advice.

> Why not just use rsync?
> 
> setup cron to just sync the two directories (/home/blah with 
> /mnt/disk/blah) over and over?

Such an arrangement will only sync /home/user on
/mnt/usb_enclosure/user at fixed time, periodically, not instaneously
whenever there is a change on /home/user

> Or just set your hom directory to /mnt/disk/user ?

I intend using the USB device for dual purpose.

1) as mirror of /home/user.  So when there is a problem there is a
second copy of data on the USB enclosure.

2) using it as mobile storge device which can be plugged in on any
Linux PC when I work on the latter.

B.R.
SL

 
> > On 8/25/06, Stephen Liu <satimis@yahoo.com> wrote:
> >>  Hi folks,
> >>
> >>  I'm prepared using an USB enclosure for data stroage.
> >>
> >>  I'm searching a tool/method whenever I save a file on /home/user
> a
> >>  duplicate copy will be saved automatically on
> >>  /mnt/point/of/usb_enclosure/home/user simultaneously, similar to
> >>  replication/mirroring.   Thus the USB enclosure can be used as a
> mobile
> >>  storage device and can be plugged in on any Linux PC.
> >
> > My first thought for this kind of setup would be a little Perl
> magic
> > with inotify... not sure if something already exists like this.
> >
> > http://www.edoceo.com/creo/inotify/perl-inotify.php
> >
> > -Mike
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Sun, 27 Aug 2006 16:55:17 +0800 (CST), Stephen Liu wrote:

> > setup cron to just sync the two directories (/home/blah with 
> > /mnt/disk/blah) over and over?  
> 
> Such an arrangement will only sync /home/user on
> /mnt/usb_enclosure/user at fixed time, periodically, not instaneously
> whenever there is a change on /home/user

Do you really need such frequent syncing? Bear in mind the limited write
lifespan of flash memory, if you have a file in your home directory that
is frequently updated, you could wear out the memory in a short time.
what's wrong with using a script that syncs your hard disk with the
flash drive before unmounting it?


-- 
Neil Bothwick

Back Up My Hard Drive? I Can't Find The Reverse Switch!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Neil,

> Do you really need such frequent syncing? Bear in mind the limited
> write
> lifespan of flash memory, if you have a file in your home directory
> that
> is frequently updated, you could wear out the memory in a short time.
> what's wrong with using a script that syncs your hard disk with the
> flash drive before unmounting it?

Mounted inside the USB enclosure is a HD.  I'll use old HD for this
purpose.  Frequently I work on at least 2 PCs running different Linux
OS.  With such a device, USB enclosure, it'll ease my work on
consolidating the working data on 2 PCs.  Otherwise I have to use a
pendrive for tranporting working data between 2 PCs.  OR compressing
them as tarball and send the latter as attachment to an email.

Tks

B.R.
SL
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Etaoin Shrdlu]

On Sunday 27 August 2006 12:20, Stephen Liu wrote:

> Mounted inside the USB enclosure is a HD.  I'll use old HD for this
> purpose.  Frequently I work on at least 2 PCs running different Linux
> OS.  With such a device, USB enclosure, it'll ease my work on
> consolidating the working data on 2 PCs.  Otherwise I have to use a
> pendrive for tranporting working data between 2 PCs.  OR compressing
> them as tarball and send the latter as attachment to an email.

If you have networking, why not set up a network share of some kind (nfs, 
smb, or even ssh via shfs) and mount it (under the same path) from each 
computer you work on? This avoid having to replicate data and, with 
decent network speed, is nearly as fast as disk I/O.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Etaoin,

Tks for your advice.

> If you have networking, why not set up a network share of some kind
> (nfs, 
> smb, or even ssh via shfs) and mount it (under the same path) from
> each 
> computer you work on? This avoid having to replicate data and, with 
> decent network speed, is nearly as fast as disk I/O.

The PCs, a workstation only, are not connected to network.  Neither
I'll run 2/3 PCs simultaneously.

Previously I did it in this way making use of an addtional network card
connecting 2 PCs.  Later I ceased using this method because another PC
was standing idly.

B.R.
satimis
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Etaoin Shrdlu]

On Sunday 27 August 2006 13:39, Stephen Liu wrote:

> The PCs, a workstation only, are not connected to network.  Neither
> I'll run 2/3 PCs simultaneously.

Ok, I didn't have this info.

> Previously I did it in this way making use of an addtional network
> card connecting 2 PCs.  Later I ceased using this method because
> another PC was standing idly.

If you don't need real-time synchronization, and don't handle gigabyes of 
data, maybe aliasing umount could be an option (this is kind of hackish, 
of course):

-------------------------------------------------------------------
#!/bin/bash

# assuming your removabile device is mounted under /mnt/external
# assuming umount is called by mountpoint and not by device name

MOUNTPOINT="/mnt/external"

if [ "$1" == "$MOUNTPOINT" ]; then

  echo "Synchronizing..."

  # put here whatever command you want to use for synching, eg rsync

  ....

fi

/bin/umount $1
exit $?
-------------------------------------------------------------------


Save tha above script into a file named "umount", and put it in your path 
so that it is found before the real /bin/umount. Or you can do the same 
with a shell alias.
Great care must be taken when modifying or aliasing important system 
commands, since you kind of break the integrity of your system.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

On Sun, 27 Aug 2006 18:20:11 +0800 (CST), Stephen Liu wrote:

> Mounted inside the USB enclosure is a HD.  I'll use old HD for this
> purpose.  Frequently I work on at least 2 PCs running different Linux
> OS.  With such a device, USB enclosure, it'll ease my work on
> consolidating the working data on 2 PCs.

OK, so you don't need to worry about write wear, but I still don't see
why you need real-time mirroring. A script to sync the drive before
unmounting would give you an accurate mirror. A cron script to check if
the drive is mounted and sync it every hour would cut down on the amount
of data to sync when unmounting.


-- 
Neil Bothwick

I heard someone tried the monkeys-on-typewriters bit trying for the plays
of W. Shakespeare but all they got was the collected works of Francis
Bacon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Neil,

> but I still don't
> see
> why you need real-time mirroring. A script to sync the drive before
> unmounting would give you an accurate mirror. A cron script to check
> if
> the drive is mounted and sync it every hour would cut down on the
> amount
> of data to sync when unmounting.

If running sync periodically on poweroff the PC there will be some
files not yet updated.  

I think there may be an existing tool for this purpose.  I believe
seeing similar tool for M$Windows previously.

B.R.
SL
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

On Sun, 27 Aug 2006 22:14:25 +0800 (CST), Stephen Liu wrote:

> > but I still don't
> > see
> > why you need real-time mirroring. A script to sync the drive before
> > unmounting would give you an accurate mirror. A cron script to check
> > if
> > the drive is mounted and sync it every hour would cut down on the
> > amount
> > of data to sync when unmounting.  
> 
> If running sync periodically on poweroff the PC there will be some
> files not yet updated.  

Shutdown involves unmounting the drive, and I recommended syncing before
unmounting.


-- 
Neil Bothwick

Modulation in all things.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Neil,

> Shutdown involves unmounting the drive, and I recommended syncing
> before unmounting.

The USB enclosure will be automatically mounted/umounted.

Whether on the script includes syncing once before poweroff/reboot
starts.

Where can I find similar syncing and cron script examples?  TIA

B.R.
SL
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Rumen Yotov]

On Sun, 27 Aug 2006 23:26:57 +0800 (CST)
Stephen Liu <satimis@yahoo.com> wrote:

> Hi Neil,
> 
> > Shutdown involves unmounting the drive, and I recommended syncing
> > before unmounting.
> 
> The USB enclosure will be automatically mounted/umounted.
> 
> Whether on the script includes syncing once before poweroff/reboot
> starts.
> 
> Where can I find similar syncing and cron script examples?  TIA
> 
> B.R.
> SL
Hi,
Not very sure that's what you want, but check: sys-cluster/drbd.
Description:         mirror/replicate block-devices across a
network-connection (requires a network connection though).
HTH.Rumen
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Rumen,

Tks for your advice.

> Not very sure that's what you want, but check: sys-cluster/drbd.
> Description:         mirror/replicate block-devices across a
> network-connection (requires a network connection though).

sync+cron can do the job, incremental-copying files from /home/user on
to /mnt/USB_enclosure/user at fixed time, periodically.  I'm now
searching for examples of sync and cron scripts to have a concept how
to start writing such scripts

B.R.
SL

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 267 bytes --]

On Sun, 27 Aug 2006 23:26:57 +0800 (CST), Stephen Liu wrote:

> Where can I find similar syncing and cron script examples?  TIA

In the rsync man page.

rsync -ax ~/ /mnt/extdisk/myhome/

should do it.


-- 
Neil Bothwick

RAM = Rarely Adequate Memory

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Neil,

> In the rsync man page.
> 
> rsync -ax ~/ /mnt/extdisk/myhome/
> 
> should do it.

On which file of cron shall I add following line to rsync
/mnt/extdisk/myhome/ quarterly?

0,15,30,45 * * * * rsync -ax ~/ /mnt/extdisk/myhome/


Whether -u update will do incremental copying for those files having
been updated

0,15,30,45 * * * * rsync -aux ~/ /mnt/extdisk/myhome/


What about those files have been deleted on ~/ ?  Which option should
be up to delete the same files on /mnt/extdisk/myhome/ simultaneously.

TIA

B.R.
SL
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Valerio daelli]

Try to add --delete.

Valerio Daelli

On 8/28/06, Stephen Liu <satimis@yahoo.com> wrote:
> Hi Neil,
>
> > In the rsync man page.
> >
> > rsync -ax ~/ /mnt/extdisk/myhome/
> >
> > should do it.
>
> On which file of cron shall I add following line to rsync
> /mnt/extdisk/myhome/ quarterly?
>
> 0,15,30,45 * * * * rsync -ax ~/ /mnt/extdisk/myhome/
>
>
> Whether -u update will do incremental copying for those files having
> been updated
>
> 0,15,30,45 * * * * rsync -aux ~/ /mnt/extdisk/myhome/
>
>
> What about those files have been deleted on ~/ ?  Which option should
> be up to delete the same files on /mnt/extdisk/myhome/ simultaneously.
>
> TIA
>
> B.R.
> SL
> --
> gentoo-user@gentoo.org mailing list
>
>
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Valerio,

> Try to add --delete.

Could you please provide more detail?  TIA

B.R.
SL
 
> On 8/28/06, Stephen Liu <satimis@yahoo.com> wrote:
> > Hi Neil,
> >
> > > In the rsync man page.
> > >
> > > rsync -ax ~/ /mnt/extdisk/myhome/
> > >
> > > should do it.
> >
> > On which file of cron shall I add following line to rsync
> > /mnt/extdisk/myhome/ quarterly?
> >
> > 0,15,30,45 * * * * rsync -ax ~/ /mnt/extdisk/myhome/
> >
> >
> > Whether -u update will do incremental copying for those files
> having
> > been updated
> >
> > 0,15,30,45 * * * * rsync -aux ~/ /mnt/extdisk/myhome/
> >
> >
> > What about those files have been deleted on ~/ ?  Which option
> should
> > be up to delete the same files on /mnt/extdisk/myhome/
> simultaneously.
> >
> > TIA
> >
> > B.R.
> > SL
> > --
> > gentoo-user@gentoo.org mailing list
> >
> >
> -- 
> gentoo-user@gentoo.org mailing list
> 
> 

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 348 bytes --]

On Mon, 28 Aug 2006 23:01:54 +0800 (CST), Stephen Liu wrote:

> Could you please provide more detail?  TIA

It's all in the rsync man page. Read that, then come back with any
questions still unanswered.


-- 
Neil Bothwick

The nice thing about Windows is - It does not just crash,
it displays adialog box and lets you press OK first.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] File replication

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 315 bytes --]

On Mon, 28 Aug 2006 19:09:11 +0800 (CST), Stephen Liu wrote:

> Whether -u update will do incremental copying for those files having
> been updated

That's not what -u does, rsync does incremental copying by default, read
the man page.


-- 
Neil Bothwick

Bald spot? No -- solar panel for brain power

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] File replication

[Roger Luethi]

On Sun, 27 Aug 2006 18:20:11 +0800, Stephen Liu wrote:
> Mounted inside the USB enclosure is a HD.  I'll use old HD for this
> purpose.  Frequently I work on at least 2 PCs running different Linux
> OS.  With such a device, USB enclosure, it'll ease my work on
> consolidating the working data on 2 PCs.  Otherwise I have to use a

You don't want to use rsync -- it is meant to copy from one master to
mirrors (like we do with portage). If you want to synchronize (i.e. update
to the latest version, no matter where it is), try unison.

Roger
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Stephen Liu]

Hi Roger,

Tks for your advice.

> If you want to synchronize (i.e.
> update
> to the latest version, no matter where it is), try unison.

Whether you suggested;
Unison
File Synchronizer
http://www.cis.upenn.edu/~bcpierce/unison/

I'm trying to find out whether real time synchronization is possible,

Tks.

B.R.
SL
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] File replication

[Roger Luethi]

On Mon, 28 Aug 2006 15:26:09 +0800, Stephen Liu wrote:
> > If you want to synchronize (i.e. update
> > to the latest version, no matter where it is), try unison.
> 
> Whether you suggested;
> Unison
> File Synchronizer
> http://www.cis.upenn.edu/~bcpierce/unison/

Yes. One of the more underestimated tools, presumably due to rsync's
popularity. It's in portage, so "emerge unison" should work.

> I'm trying to find out whether real time synchronization is possible,

unison in batch mode (-batch) triggered by umount (as suggested by others)
should be close enough.

Roger
-- 
gentoo-user@gentoo.org mailing list