[gentoo-user] Symlinked directories and rsync

[gentoo-user] Symlinked directories and rsync

[Walter Dnes]

  I've RTFM'd, and it looks like there is no way to get rsync to copy
over symlinked directories as symlinks.  E.g. I build Pale Moon browser
from source for 3 machines at home.  I prefer downloading the source
once and symlinking to it from 3 different working directories.  I also
have a personal project where I analyze the same text data files in 3
different ways.  It makes things so much simpler to use the same code,
with different runtime parameters.  So I symlink the main data directory
to 3 working directories.  Even "better" (or worse) I've got a situation
where I symlink directories to... other symlinked directories.  Yes;
symlinks pointing to symlinks work, even for directories.

  Unfortunately, rsync will not create a symlink where appropriate, but
will de-reference the symlink, and duplicate multiple gigabytes of data
needlessly.  I eventually gave up trying, and made a tarball, and pushed
it over to my "hot backup" machine, and extracted it there.  That works
OK for the initial setup, but is painfull for "minor incremental"
updates.

  Am I missing something glaringly obvious, or is this a limitation of
rsync?  If so, is there another tool that can copy over symlinked
directories properly?

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Symlinked directories and rsync

[John Covici]

On Sat, 19 Aug 2017 00:56:55 -0400,
Walter Dnes wrote:
> 
>   I've RTFM'd, and it looks like there is no way to get rsync to copy
> over symlinked directories as symlinks.  E.g. I build Pale Moon browser
> from source for 3 machines at home.  I prefer downloading the source
> once and symlinking to it from 3 different working directories.  I also
> have a personal project where I analyze the same text data files in 3
> different ways.  It makes things so much simpler to use the same code,
> with different runtime parameters.  So I symlink the main data directory
> to 3 working directories.  Even "better" (or worse) I've got a situation
> where I symlink directories to... other symlinked directories.  Yes;
> symlinks pointing to symlinks work, even for directories.
> 
>   Unfortunately, rsync will not create a symlink where appropriate, but
> will de-reference the symlink, and duplicate multiple gigabytes of data
> needlessly.  I eventually gave up trying, and made a tarball, and pushed
> it over to my "hot backup" machine, and extracted it there.  That works
> OK for the initial setup, but is painfull for "minor incremental"
> updates.
> 
>   Am I missing something glaringly obvious, or is this a limitation of
> rsync?  If so, is there another tool that can copy over symlinked
> directories properly?

I always use -av which copies simlinks correctly.  -H is necessary to
copy hardlinks.  If you need it -l also ensures this.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] Symlinked directories and rsync

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1843 bytes --]

On Saturday, 19 August 2017 06:33:27 BST John Covici wrote:
> On Sat, 19 Aug 2017 00:56:55 -0400,
> 
> Walter Dnes wrote:
> >   I've RTFM'd, and it looks like there is no way to get rsync to copy
> > 
> > over symlinked directories as symlinks.  E.g. I build Pale Moon browser
> > from source for 3 machines at home.  I prefer downloading the source
> > once and symlinking to it from 3 different working directories.  I also
> > have a personal project where I analyze the same text data files in 3
> > different ways.  It makes things so much simpler to use the same code,
> > with different runtime parameters.  So I symlink the main data directory
> > to 3 working directories.  Even "better" (or worse) I've got a situation
> > where I symlink directories to... other symlinked directories.  Yes;
> > symlinks pointing to symlinks work, even for directories.
> > 
> >   Unfortunately, rsync will not create a symlink where appropriate, but
> > 
> > will de-reference the symlink, and duplicate multiple gigabytes of data
> > needlessly.  I eventually gave up trying, and made a tarball, and pushed
> > it over to my "hot backup" machine, and extracted it there.  That works
> > OK for the initial setup, but is painfull for "minor incremental"
> > updates.
> > 
> >   Am I missing something glaringly obvious, or is this a limitation of
> > 
> > rsync?  If so, is there another tool that can copy over symlinked
> > directories properly?
> 
> I always use -av which copies simlinks correctly.  -H is necessary to
> copy hardlinks.  If you need it -l also ensures this.

I've also found that 'rsync -a -l' does the job, but other tools exist to 
achieve the same:

https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html

https://linux.die.net/man/1/star (look for the section 'Incremental Backups'

HTH.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: Symlinked directories and rsync

[Ian Zimmerman]

On 2017-08-19 09:59, Mick wrote:

> > > Am I missing something glaringly obvious, or is this a limitation
> > > of rsync?  If so, is there another tool that can copy over
> > > symlinked directories properly?
> > 
> > I always use -av which copies simlinks correctly.  -H is necessary
> > to copy hardlinks.  If you need it -l also ensures this.
> 
> I've also found that 'rsync -a -l' does the job, but other tools exist
> to achieve the same:
> 
> https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html
> 
> https://linux.die.net/man/1/star (look for the section 'Incremental Backups'

For the purposes about which Walter asks, I do not symlink directories,
I symlink the files.  IOW, I create what is known as "symlink farms".
There are also multiple tools for doing that:

1.  lndir, in the x11-misc/lndir package on gentoo

2.  on a GNU system, cp -rsT

3.  symlink-tree script in older releases of GNU automake (on my gentoo
    system, in the sys-devel/automake-1.11.6-r1 package)

-- 
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
Do obvious transformation on domain to reply privately _only_ on Usenet.

[gentoo-user] Re: Symlinked directories and rsync

[Ian Zimmerman]

On 2017-08-19 09:41, Ian Zimmerman wrote:

> For the purposes about which Walter asks, I do not symlink directories,
> I symlink the files.  IOW, I create what is known as "symlink farms".
> There are also multiple tools for doing that:
> 
> 1.  lndir, in the x11-misc/lndir package on gentoo
> 
> 2.  on a GNU system, cp -rsT
> 
> 3.  symlink-tree script in older releases of GNU automake (on my gentoo
>     system, in the sys-devel/automake-1.11.6-r1 package)

After I wrote that, I had the nagging thought that none of those in fact
worked just as I wanted, and worse, they were Not Invented Here (TM).
Thus, I proceeded to write my own:

https://gist.github.com/nobrowser/dfeb275f3273d1e3887c2a24e6f596a6

-- 
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
Do obvious transformation on domain to reply privately _only_ on Usenet.

Re: [gentoo-user] Symlinked directories and rsync

[Walter Dnes]

On Sat, Aug 19, 2017 at 09:59:47AM +0100, Mick wrote

> I've also found that 'rsync -a -l' does the job, but other tools
> exist to achieve the same:

  Thanks; I'll try that next time I do an rsync to the "hot backup"
machine.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications