[gentoo-user] sync'ing two computers (not related to emerge)

[gentoo-user] sync'ing two computers (not related to emerge)

[Simon]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

hi there!
  I've been using unison to synchronise and backup my computers.  unison is
based on rsync IIRC but with the difference that it 'remembers' the state of
the folder that was synchronised.  This way, if I delete a file on A, when
sync'ing it will be deleted on B.  While rsync would leave the file on B...

  I see rsync as a very good 'cp' program, but I need something that can
deal with conflicts (file changed on both hosts/directories) and can
propagate changes like deletes properly.  Is there a way to get this
behavior with rsync (maybe i missed something)?

  The reason i'm asking is that unison has been having problems recently in
sync'ing two pc...  On my laptop, when i start the sync, it first checks it
for changes against the 'unison snapshot'.  Changes done since last sync are
those that will be propagated.  But it takes so long to check for changes
(disk input rate is too slow i think) that once it finished, when it's time
to send the changes to the remote server, the remote end has already
disconnected!

Thanks for the help/suggestions!

[-- Attachment #2: Type: text/html, Size: 1147 bytes --]

Re: [gentoo-user] sync'ing two computers (not related to emerge)

[Alan McKinnon]

On Saturday 02 May 2009 23:33:38 Simon wrote:
> hi there!
>   I've been using unison to synchronise and backup my computers.  unison is
> based on rsync IIRC but with the difference that it 'remembers' the state
> of the folder that was synchronised.  This way, if I delete a file on A,
> when sync'ing it will be deleted on B.  While rsync would leave the file on
> B...

I assume you are syncing from A to B? In that case you should look at the 
various --delete* options to rsync 

>   I see rsync as a very good 'cp' program, but I need something that can
> deal with conflicts (file changed on both hosts/directories) and can
> propagate changes like deletes properly.  Is there a way to get this
> behavior with rsync (maybe i missed something)?

This cannot be done with rsync. It's very much an efficient scp replacement 
that works with deltas. And there is a definite source and destination, what 
you are describing is like how source repos work

>   The reason i'm asking is that unison has been having problems recently in
> sync'ing two pc...  On my laptop, when i start the sync, it first checks it
> for changes against the 'unison snapshot'.  Changes done since last sync
> are those that will be propagated.  But it takes so long to check for
> changes (disk input rate is too slow i think) that once it finished, when
> it's time to send the changes to the remote server, the remote end has
> already disconnected!

Surely unison has a --timeout or similar option?
rsync has it, and it's such an obviously needed config option


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] sync'ing two computers (not related to emerge)

[Brandon Vargo]

On Sat, 2009-05-02 at 17:33 -0400, Simon wrote:
> hi there!
>   I've been using unison to synchronise and backup my computers.
> unison is based on rsync IIRC but with the difference that it
> 'remembers' the state of the folder that was synchronised.  This way,
> if I delete a file on A, when sync'ing it will be deleted on B.  While
> rsync would leave the file on B...
> 
>   I see rsync as a very good 'cp' program, but I need something that
> can deal with conflicts (file changed on both hosts/directories) and
> can propagate changes like deletes properly.  Is there a way to get
> this behavior with rsync (maybe i missed something)?

No. Unison implements the rsync protocol for transferring files, but it
does not actually use rsync itself. You can make rsync transfer files
that are not in either the destination or a "reference" location with
the --compare-dest option, for example if doing an incremental backup,
but this will not propagate the deletion of a file or directory. The
delete file will come back on the next synchronization.

>   The reason i'm asking is that unison has been having problems
> recently in sync'ing two pc...  On my laptop, when i start the sync,
> it first checks it for changes against the 'unison snapshot'.  Changes
> done since last sync are those that will be propagated.  But it takes
> so long to check for changes (disk input rate is too slow i think)
> that once it finished, when it's time to send the changes to the
> remote server, the remote end has already disconnected!
> 
> Thanks for the help/suggestions!

If you are using ssh with unison, are you using the "KeepAlive yes"
option in your ssh configuration? If not, add it, and your connection
should not close from inactivity. If you are using direct sockets,
unison will use a keepalive so it can timeout if the communication link
is broken.

How long is unison taking to check for changes? I can usually reconcile
changes in my home directory (approximately 45G, 125,000 files) in less
than 10-15 seconds between my slow laptop drive and a remote machine.
However, if you are syncing with a Windows machine, expect the sync to
be much slower. Fast checking is not safe on Windows, as unison can miss
changes, so it scans every file every time unless you tell it otherwise.

Regards,

Brandon Vargo

Re: [gentoo-user] sync'ing two computers (not related to emerge)

[Simon]

> If you are using ssh with unison, are you using the "KeepAlive yes"
> option in your ssh configuration? If not, add it, and your connection
> should not close from inactivity. If you are using direct sockets,
> unison will use a keepalive so it can timeout if the communication link
> is broken.

Well i was using it, but i found out the option was commented out on
my host's config...  i set it to yes and trying again now...  will
have results in a few mins....

> How long is unison taking to check for changes? I can usually reconcile
> changes in my home directory (approximately 45G, 125,000 files) in less
> than 10-15 seconds between my slow laptop drive and a remote machine.
> However, if you are syncing with a Windows machine, expect the sync to
> be much slower. Fast checking is not safe on Windows, as unison can miss
> changes, so it scans every file every time unless you tell it otherwise.

Well, the problem seems to be with my /usr/portage that had not been
sync'ed for a long time (i guess it has to make thorough difference
checking?).  So there is lots of files to check.  It really takes
minutes to scan for changes, i'm timing this run, if it fails i'll
post how long it takes to do the whole scan until it starts
transfering and finds out it got timed out.
Sync'ing is done from linux to linux.  The problem is really hardware
related; i'm thinking on getting a new portable HD, but since i'm
about to loose my job,i might want to wait!  ;)

Thanks!

Re: [gentoo-user] sync'ing two computers (not related to emerge)

[Simon]

> How long is unison taking to check for changes? I can usually reconcile
> changes in my home directory (approximately 45G, 125,000 files) in less
> than 10-15 seconds between my slow laptop drive and a remote machine.
> However, if you are syncing with a Windows machine, expect the sync to
> be much slower. Fast checking is not safe on Windows, as unison can miss
> changes, so it scans every file every time unless you tell it otherwise.

Ok i had to try a few times and now with tcpkeepalive being turned on
it doesnt disconnect.  But when the scan is done, it 'reconciles' the
changes (so it has received the changes from the remote end) and
Begins the transfer of a dozen files, but not a bit is transfered.
The sshd and unison process on the remote host are waiting/sleeping or
in coma, but nothing happens.

I just tried again and let it sit on the transfer to see if the
processes would wake up, but it read from remote host that connection
timed out.  I'm not sure but this may be due to the fact i'm on an
unstable link... I'll try once more with a ssh session on the side to
see if that session will get disconnected or not...

It would be so nice if unison could 'remember' the changes on my HD
against its local archive...  anyway...

Thanks, Simon

Re: [gentoo-user] sync'ing two computers (not related to emerge)

[Brandon Vargo]

On Sun, 2009-05-03 at 10:32 -0400, Simon wrote:
> > How long is unison taking to check for changes? I can usually reconcile
> > changes in my home directory (approximately 45G, 125,000 files) in less
> > than 10-15 seconds between my slow laptop drive and a remote machine.
> > However, if you are syncing with a Windows machine, expect the sync to
> > be much slower. Fast checking is not safe on Windows, as unison can miss
> > changes, so it scans every file every time unless you tell it otherwise.
> 
> Ok i had to try a few times and now with tcpkeepalive being turned on
> it doesnt disconnect.  But when the scan is done, it 'reconciles' the
> changes (so it has received the changes from the remote end) and
> Begins the transfer of a dozen files, but not a bit is transfered.
> The sshd and unison process on the remote host are waiting/sleeping or
> in coma, but nothing happens.
> 
> I just tried again and let it sit on the transfer to see if the
> processes would wake up, but it read from remote host that connection
> timed out.  I'm not sure but this may be due to the fact i'm on an
> unstable link... I'll try once more with a ssh session on the side to
> see if that session will get disconnected or not...
> 
> It would be so nice if unison could 'remember' the changes on my HD
> against its local archive...  anyway...
> 
> Thanks, Simon

An unstable link is probably triggering your issue. If the link dies and
comes back, unison (at least version 2.27.57, what I have here) gets
confused and just sits there. What's interesting is that if unison is
killed on the initiating machine, the remote process stops as well,
which leads me to believe this is a bug in unison; they can talk, but
they don't do anything. I've glanced at the code before despite my lack
of familiarity with OCaml, as my wireless connection will often die,
causing much grief, but I've never found a solution.

You could try syncing subdirectories iteratively within what you want to
sync and hope that your connection does not die during the smaller sync,
but that really isn't a good solution. I personally use this method to
speed up synchronization if, for example, I only need my latest CS
project synchronized but not any new music I may have. Unison is smart
enough to use the same hash database if you use the same root
directories and use the Path directive in your configuration to limit
what is synchronized, so it should not re-hash your files on a later
sync.

As for other solutions, if you cannot get unison to work with your
connection, I would suggest using a DVCS. I tried synchronizing my home
directory with git before finding unison. The DVCS approach is better
than SVN or CVS because it takes up less space and does not require a
server. The disadvantages are the increased space used to store the
backend files and the need to commit all of your files before
pushing/pulling to/from the other machine (though I guess that could be
an advantage, depending on your needs). It isn't optimal, but it can
work, despite the git manual specifically mentioning that git was not
made to do such a thing. Mercurial might work better, but I've never
tried to use it. You might hit the same issues if the connection dies; I
don't know.

If you do find something that works similar to unison, but is more
robust, I'd love to hear about it. I've tired a number of things, but I
keep coming back to unison, despite its issues.

Regards,

Brandon Vargo

Re: [gentoo-user] sync'ing two computers (not related to emerge)

[Simon]

Issue is resolved (solution below)

> An unstable link is probably triggering your issue. If the link dies and
> comes back, unison (at least version 2.27.57, what I have here) gets
> confused and just sits there. What's interesting is that if unison is
> killed on the initiating machine, the remote process stops as well,
> which leads me to believe this is a bug in unison; they can talk, but
> they don't do anything. I've glanced at the code before despite my lack
> of familiarity with OCaml, as my wireless connection will often die,
> causing much grief, but I've never found a solution.

The problem was actually related to ssh...  setting those in
sshd_config fixed it:
ClientAliveInterval 15
TCPKeepAlive no
---
The TCPKeepAlive option set to yes doesnt seem sufficient, it was
suggested by someone to use ClientAliveInterval instead.  (or on the
client end i think you could set ServerAliveInterval)

Thanks all!
Simon

Re: [gentoo-user] sync'ing two computers (not related to emerge)

[Stroller]

[-- Attachment #1: Type: text/plain, Size: 715 bytes --]


On 2 May 2009, at 22:33, Simon wrote:
> ...
>   I've been using unison to synchronise and backup my computers.   
> unison is based on rsync IIRC but with the difference that it  
> 'remembers' the state of the folder that was synchronised.  This  
> way, if I delete a file on A, when sync'ing it will be deleted on  
> B.  While rsync would leave the file on B...
>
>   I see rsync as a very good 'cp' program, but I need something that  
> can deal with conflicts (file changed on both hosts/directories) and  
> can propagate changes like deletes properly.  ...


http://www.linuxjournal.com/article/5976
http://kitenet.net/~joey/svnhome/

http://www.google.com/search?q=homedir+version+control

Stroller.





[-- Attachment #2: Type: text/html, Size: 1252 bytes --]

Re: [gentoo-user] sync'ing two computers (not related to emerge)

[Simon]

> http://www.linuxjournal.com/article/5976
> http://kitenet.net/~joey/svnhome/
> http://www.google.com/search?q=homedir+version+control

Hey thanks Stroller,
  I'd be interested in these solutions, but i dont have enough space
to copy everything in double (keep a backup).  So what i do, and this
is why i need unison, is that my older computer is the backup of my
newer computer.  Only a few files here and there are not copied over
(those that are computer specific).

  But your idea was very good and I'll keep note of it... I'm thinking
on buying a hard drive as soon as it becomes possible (unfortunately
in my situation, this may not happen until a few more months) and when
i get that, I'll set it up as a central repository (if i havent' found
any better solution until then).

Thanks!