[gentoo-user] What utility do you use to sync user files?

[gentoo-user] What utility do you use to sync user files?

[Randy Westlund]

I've been using rsync to sync binary files, shell scripts, my
workspace, and random user files under my home directory across
multiple machines.  I'm using one server as the master copy, which
makes daily incremental backups of my files to a separate disk with
rsync.  At the moment, I have my sync script set up as a Makefile with
the following targets.  I run this from multiple workstations.

It would be nice to use something as easy as svn, but many of my files
are binary.  Or something like dropbox would be great.  I don't work
from windows, so I don't need a cross-platform solution.

What utilities do you guys use?  Is there a better way to do this?  It
would be nice to move everything to the background, but I've already
clobbered a few files by calling this in the wrong order and might
move the Makefile to an interactive script to protect against that.  I
have to call 'make clobber' after I remove a local file to push that
change to the server, and if I forgot to call 'make get' first, I have
to fix it manually.

-------sync makefile--------
get:
        rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
        --delete \
        $(HOST):$(SERVER_DIR) $(LOCAL_DIR)

put:
        rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
        $(LOCAL_DIR) $(HOST):$(SERVER_DIR)

clobber:
        rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
        --delete \
        $(LOCAL_DIR) $(HOST):$(SERVER_DIR)
------end-------

-------backup script--------
# if files are already there, hard link
# the last lines mark it as complete and move a soft link pointer
rsync -zavi --progress --delete \
  --link-dest=$BACKUP_PATH/current \
  $SOURCE $BACKUP_PATH/backup_part_$DATE \
  && mv $BACKUP_PATH/backup_part_$DATE $BACKUP_PATH/backup_$DATE \
  && unlink $BACKUP_PATH/current \
  && ln -s $BACKUP_PATH/backup_$DATE $BACKUP_PATH/current
-------end---------

Randy

Re: [gentoo-user] What utility do you use to sync user files?

[Mark Knecht]

On Sun, Dec 2, 2012 at 9:21 AM, Randy Westlund <rwestlun@gmail.com> wrote:
> I've been using rsync to sync binary files, shell scripts, my
> workspace, and random user files under my home directory across
> multiple machines.  I'm using one server as the master copy, which
> makes daily incremental backups of my files to a separate disk with
> rsync.  At the moment, I have my sync script set up as a Makefile with
> the following targets.  I run this from multiple workstations.
>
> It would be nice to use something as easy as svn, but many of my files
> are binary.  Or something like dropbox would be great.  I don't work
> from windows, so I don't need a cross-platform solution.
>
> What utilities do you guys use?  Is there a better way to do this?  It
> would be nice to move everything to the background, but I've already
> clobbered a few files by calling this in the wrong order and might
> move the Makefile to an interactive script to protect against that.  I
> have to call 'make clobber' after I remove a local file to push that
> change to the server, and if I forgot to call 'make get' first, I have
> to fix it manually.
>

Dunno if it's of any value but the new Linux Journal has a pointer to
grsync which is in portage and built fine on my system here. It looks
interesting in that you can create 'sessions' which probably could do
much of what at least I'm doing in simple rsync scripts. I'm going to
play with it a bit more but thought I'd point it out as I hadn't heard
of it before.

HTH,
Mark

Re: [gentoo-user] What utility do you use to sync user files?

[Bruce Hill]

On Sun, Dec 02, 2012 at 12:21:40PM -0500, Randy Westlund wrote:
> I've been using rsync to sync binary files, shell scripts, my
> workspace, and random user files under my home directory across
> multiple machines.  I'm using one server as the master copy, which
> makes daily incremental backups of my files to a separate disk with
> rsync.  At the moment, I have my sync script set up as a Makefile with
> the following targets.  I run this from multiple workstations.
> 
> It would be nice to use something as easy as svn, but many of my files
> are binary.  Or something like dropbox would be great.  I don't work
> from windows, so I don't need a cross-platform solution.
> 
> What utilities do you guys use?  Is there a better way to do this?  It
> would be nice to move everything to the background, but I've already
> clobbered a few files by calling this in the wrong order and might
> move the Makefile to an interactive script to protect against that.  I
> have to call 'make clobber' after I remove a local file to push that
> change to the server, and if I forgot to call 'make get' first, I have
> to fix it manually.
> 
> -------sync makefile--------
> get:
>         rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
>         --delete \
>         $(HOST):$(SERVER_DIR) $(LOCAL_DIR)
> 
> put:
>         rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
>         $(LOCAL_DIR) $(HOST):$(SERVER_DIR)
> 
> clobber:
>         rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
>         --delete \
>         $(LOCAL_DIR) $(HOST):$(SERVER_DIR)
> ------end-------
> 
> -------backup script--------
> # if files are already there, hard link
> # the last lines mark it as complete and move a soft link pointer
> rsync -zavi --progress --delete \
>   --link-dest=$BACKUP_PATH/current \
>   $SOURCE $BACKUP_PATH/backup_part_$DATE \
>   && mv $BACKUP_PATH/backup_part_$DATE $BACKUP_PATH/backup_$DATE \
>   && unlink $BACKUP_PATH/current \
>   && ln -s $BACKUP_PATH/backup_$DATE $BACKUP_PATH/current
> -------end---------
> 
> Randy

What about Network File System (NFS)? Our little wiki has an article:
https://wiki.gentoo.org/wiki/NFSv4 -- haven't read it yet, though.
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] What utility do you use to sync user files?

[Alan McKinnon]

As soon as you said "svn" I got the idea you want revisions of files
you send to storage, which is a fine idea. Backups are fine, but if you
have to work at the granularity of "yesterday or maybe the day before"
it can become cumbersome.

To my mind the first question to answer is how do you want to commit
these files to a revision control system. This determines what you do
next. 

If you want it file-by-file, then getting a single file is a
simple svn checkout, but rolling everything back to yesterday (eg for
when you accidentally did an rm on .kde or .mail) will be tricky. To do
this, commit the files to remote svn and just put up with the hassles
of the binary files. Yes, I know binary in SVN is amazingly evil and
makes the worst ever out of Redmond look like child's play, but
sometimes you have no choice... Or use a revision control system that
can deal with binaries OK. All depends on what *you* need stuff to do.

If you are happy committing an entire Makefile run and are OK to work
with that, then you use an FS (or LVM) that can do snapshots, and make
one as the last command in your Makefile. Restores are a case of mount
the snapshot, find the file of interest and do what you need with it.

It really comes down to the old story of "precisely define first
what exactly you want to achieve" and then the tools to do that often
present themselves.


 On Sun, 2 Dec 2012 12:21:40 -0500
Randy Westlund <rwestlun@gmail.com> wrote:

> I've been using rsync to sync binary files, shell scripts, my
> workspace, and random user files under my home directory across
> multiple machines.  I'm using one server as the master copy, which
> makes daily incremental backups of my files to a separate disk with
> rsync.  At the moment, I have my sync script set up as a Makefile with
> the following targets.  I run this from multiple workstations.
> 
> It would be nice to use something as easy as svn, but many of my files
> are binary.  Or something like dropbox would be great.  I don't work
> from windows, so I don't need a cross-platform solution.
> 
> What utilities do you guys use?  Is there a better way to do this?  It
> would be nice to move everything to the background, but I've already
> clobbered a few files by calling this in the wrong order and might
> move the Makefile to an interactive script to protect against that.  I
> have to call 'make clobber' after I remove a local file to push that
> change to the server, and if I forgot to call 'make get' first, I have
> to fix it manually.
> 
> -------sync makefile--------
> get:
>         rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
>         --delete \
>         $(HOST):$(SERVER_DIR) $(LOCAL_DIR)
> 
> put:
>         rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
>         $(LOCAL_DIR) $(HOST):$(SERVER_DIR)
> 
> clobber:
>         rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
>         --delete \
>         $(LOCAL_DIR) $(HOST):$(SERVER_DIR)
> ------end-------
> 
> -------backup script--------
> # if files are already there, hard link
> # the last lines mark it as complete and move a soft link pointer
> rsync -zavi --progress --delete \
>   --link-dest=$BACKUP_PATH/current \
>   $SOURCE $BACKUP_PATH/backup_part_$DATE \
>   && mv $BACKUP_PATH/backup_part_$DATE $BACKUP_PATH/backup_$DATE \
>   && unlink $BACKUP_PATH/current \
>   && ln -s $BACKUP_PATH/backup_$DATE $BACKUP_PATH/current
> -------end---------
> 
> Randy
> 



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] What utility do you use to sync user files?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Sun, 2 Dec 2012 12:21:40 -0500, Randy Westlund wrote:

> What utilities do you guys use?  Is there a better way to do this?  It
> would be nice to move everything to the background, but I've already
> clobbered a few files by calling this in the wrong order

net-misc/unison

Think of it as a two-way rsync. It keeps track of what has changed and
been synced on each host and makes changes in both directions on a single
run. If you add or modify a file on one side, it copies it over. If you
delete a file on one side it deletes if from the other and if you modify
a file on both sides since the last run, it asks what you want to do (or
skips the file if running non-interactively).

-- 
Neil Bothwick

c:>Press Enter to Exit

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] What utility do you use to sync user files?

[Dustin C. Hatch]

On 12/2/2012 14:33, Neil Bothwick wrote:
> On Sun, 2 Dec 2012 12:21:40 -0500, Randy Westlund wrote:
>
>> What utilities do you guys use?  Is there a better way to do this?  It
>> would be nice to move everything to the background, but I've already
>> clobbered a few files by calling this in the wrong order
>
> net-misc/unison
>
>
I use unison to emulate Windows's "offline files" feature for several 
subdirectories in ~, and I can say it works really well. It took me 
quite some time to understand all of its options, and it has some very 
strange behavioral quirks which are easily worked around, but I like the 
what I ended up with. I currently have it set up to automatically 
synchronize a couple of locations on my notebook with a share on my file 
server about every five minutes. If the server is unavailable (i.e. I am 
not at home), it exits silently, but will try again at the next 
scheduled time. In the event of file conflicts (i.e. I changed the same 
file on the server and on the notebook between syncs), it sends me an 
email listing the conflicting filenames, and I can look into it later.

-- 
♫Dustin

Re: [gentoo-user] What utility do you use to sync user files?

[Randy Westlund]

On Sun, Dec 2, 2012 at 10:14 PM, Dustin C. Hatch <admiralnemo@gmail.com> wrote:
> On 12/2/2012 14:33, Neil Bothwick wrote:
>>
>> On Sun, 2 Dec 2012 12:21:40 -0500, Randy Westlund wrote:
>>
>>> What utilities do you guys use?  Is there a better way to do this?  It
>>> would be nice to move everything to the background, but I've already
>>> clobbered a few files by calling this in the wrong order
>>
>>
>> net-misc/unison
>>
>>
> I use unison to emulate Windows's "offline files" feature for several
> subdirectories in ~, and I can say it works really well. It took me quite
> some time to understand all of its options, and it has some very strange
> behavioral quirks which are easily worked around, but I like the what I
> ended up with. I currently have it set up to automatically synchronize a
> couple of locations on my notebook with a share on my file server about
> every five minutes. If the server is unavailable (i.e. I am not at home), it
> exits silently, but will try again at the next scheduled time. In the event
> of file conflicts (i.e. I changed the same file on the server and on the
> notebook between syncs), it sends me an email listing the conflicting
> filenames, and I can look into it later.
>
> --
> ♫Dustin
>


Unison is exactly what I was looking for.  Thanks, Neil and Dustin.

Randy

Re: [gentoo-user] What utility do you use to sync user files?

[Joseph]

Here is a good example on using "rsync" 
http://rsync.samba.org/examples.html

I've modified the first one for 7-days incremental backup.

--
Joseph

On 12/02/12 12:21, Randy Westlund wrote:
>I've been using rsync to sync binary files, shell scripts, my
>workspace, and random user files under my home directory across
>multiple machines.  I'm using one server as the master copy, which
>makes daily incremental backups of my files to a separate disk with
>rsync.  At the moment, I have my sync script set up as a Makefile with
>the following targets.  I run this from multiple workstations.
>
>It would be nice to use something as easy as svn, but many of my files
>are binary.  Or something like dropbox would be great.  I don't work
>from windows, so I don't need a cross-platform solution.
>
>What utilities do you guys use?  Is there a better way to do this?  It
>would be nice to move everything to the background, but I've already
>clobbered a few files by calling this in the wrong order and might
>move the Makefile to an interactive script to protect against that.  I
>have to call 'make clobber' after I remove a local file to push that
>change to the server, and if I forgot to call 'make get' first, I have
>to fix it manually.
>
>-------sync makefile--------
>get:
>        rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
>        --delete \
>        $(HOST):$(SERVER_DIR) $(LOCAL_DIR)
>
>put:
>        rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
>        $(LOCAL_DIR) $(HOST):$(SERVER_DIR)
>
>clobber:
>        rsync -azOuvihh --progress -e ssh $(EXCLUDE) \
>        --delete \
>        $(LOCAL_DIR) $(HOST):$(SERVER_DIR)
>------end-------
>
>-------backup script--------
># if files are already there, hard link
># the last lines mark it as complete and move a soft link pointer
>rsync -zavi --progress --delete \
>  --link-dest=$BACKUP_PATH/current \
>  $SOURCE $BACKUP_PATH/backup_part_$DATE \
>  && mv $BACKUP_PATH/backup_part_$DATE $BACKUP_PATH/backup_$DATE \
>  && unlink $BACKUP_PATH/current \
>  && ln -s $BACKUP_PATH/backup_$DATE $BACKUP_PATH/current
>-------end---------
>
>Randy