[gentoo-user] rsync not deleting removed directory and drive PW DIS 3.3v question

[gentoo-user] rsync not deleting removed directory and drive PW DIS 3.3v question

[Dale]

Howdy,

I was doing a backup update and noticed a directory I didn't like the
name of, forgot to change it before starting the back up.  So, I stopped
the backup and changed the name, which is similar to removing it from
the perspective of the backup.  I then started the backup command again
and noticed it did not delete the old directory name which is no longer
on the source.  It did however copy the new directory.  I had to delete
the old directory manually.  To test this, I copied a directory with a
new name, added "-1" to the end.  I updated the backup which added the
directory with -1 as expected.  When it was done, I deleted the
directory with the -1 from the source and ran the command again.  It did
not delete the directory I removed as expected on the backup/target. 

I do this quite often with files and it seems to work as I expect.  If I
change the name of a file, it removes the file with old name, copies
over the file with the new name and moves on.  I've done that more times
than I care to count.  Why is a directory different?  Do I need to add a
option?  This is a example command but most all of them are the same
just the source/target directories change. 


time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Video/*
/mnt/10tb/Video/


I've tried removing the -u, adding --force but no change.  Basically, if
I remove a directory on the source, how do I get it to remove the same
on the backup/target?  I went through the options on the man page and
nothing hit me as a fix.  What am I missing? 

Also, I'm about to order a new hard drive.  I checked and my power
supply cables do have the 3.3v wire for SATA power connectors.  I plan
to buy a hard drive, either a 12TB or 14TB.  Is there a easy way to know
if it has that pin that disables the drive before I buy?  Most have
pictures of the drives.  I guess I could ask them but that would delay
my ordering.  I tried google.  I don't see a list of drive that have or
don't have this.  After some other searching.  I did find some manuals
for drives I'm looking at buying.  If it lists pin 1 through 3 as 3.3v,
is it the old way, works when plugged in?  If it says PW DIS or reserved
for pin 3, then it is new and will stay off when plugged in?  Am I
getting this right?  Close even?  lol 

Thanks.

Dale

:-)  :-) 

Re: [gentoo-user] rsync not deleting removed directory and drive PW DIS 3.3v question

[Arve Barsnes]

On Fri, 2 Sept 2022 at 10:36, Dale <rdalek1967@gmail.com> wrote:
> I've tried removing the -u, adding --force but no change.  Basically, if
> I remove a directory on the source, how do I get it to remove the same
> on the backup/target?  I went through the options on the man page and
> nothing hit me as a fix.  What am I missing?

Not sure if it would make any difference, but did you try the prune
empty dirs switch, '-m'?

Regards,
Arve

Re: [gentoo-user] rsync not deleting removed directory and drive PW DIS 3.3v question

[Ramon Fischer]

[-- Attachment #1.1: Type: text/plain, Size: 2168 bytes --]

Hello Dale,

if I understand you correctly, you had renamed the source directory

    "/home/dale/Desktop/Crypt/Video/" to
    "/home/dale/Desktop/Crypt/Video-1/" and

want to have it removed in the target directory, which is still

    "/mnt/10tb/Video/"?

Important!: Please read everything first, then do an isolated test in 
"/tmp/", since the commands are dangerous!:

    You can instruct "rsync" to copy the entire parent directory:

        $ rsync --delete "[...]/Crypt/" "/mnt/10tb"

    Note the trailing slash behind "Crypt", which is an indicator[1] to
    tell "rsync", that it is a directory and that all files and
    subdirectories should be copied to the target directory "/mnt/10tb".
    If something is missing in the source directory "[...]/Crypt/", it
    will be also deleted in "/mnt/10tb"!

Side notes, which may be useful:

    1. "rsync" is also capable of creating the (missing) target directory:

        $ rsync --delete "[...]/Crypt/Video-1/" "/mnt/10tb/something"

    "rsync" creates the target directory "/mnt/10tb/something" first and
    then copies all files and subdirectories from the source directory
    "[...]/Crypt/Video-1/" to it. But "/mnt/10tb/Video/" would still be
    orphaned.

    2. There is a parameter, called "--mkpath"[2], which creates all
    missing subdirectories of the target directory first.

-Ramon

[1] See "man 1 rsync" "USAGE":
"A trailing slash on the source changes this behavior to avoid creating 
an additional directory level at the destination. You can think of a 
trailing / on a source as meaning "copy the contents of this directory"[...]
[2] See "man 1 rsync".


On 02/09/2022 10:35, Dale wrote:
> time rsync -auv --progress --delete/home/dale/Desktop/Crypt/Video/*
> /mnt/10tb/Video/
>
>
> I've tried removing the -u, adding --force but no change.  Basically, if
> I remove a directory on the source, how do I get it to remove the same
> on the backup/target?  I went through the options on the man page and
> nothing hit me as a fix.  What am I missing?

-- 
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-user] rsync not deleting removed directory and drive PW DIS 3.3v question

[Sebastiaan L. Zoutendijk]

Dear Dale,

On Friday 2 September 2022, 3.35am -0500, Dale wrote:

> time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Video/*
> /mnt/10tb/Video/

What is going on here is due to a subtle interplay of rsync’s syntax and
shell wildcard expansion.  You can read about the details of rsync
syntax in its man page, but I’ll illustrate here:

rsync has two ways to specify a source directory.

1.  Without a trailing slash: `rsync <options> path1/source path2/dest`.
    This syncs path1/source -> path2/dest/source.

2.  With a trailing slash: `rsync <options> path1/source/ path2/dest`.
    This syncs path1/source -> path2/dest.

Note that the two ways lead to different target directories getting
synced with the source directory.  I think the command you are looking
for is therefore

time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Video /mnt/10tb

or

time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Video/
/mnt/10tb/Video

NOTE: I have not tested these commands, I would advise you to run them
with --dry-run first.

Now, why is your original command not deleting the old subdirectory?
Let me illustrate this with another example.  Suppose we have the same
directories as in my first example, path1/source and path2/dest.  Now
suppose path1/source contains subdirectories foo and bar.  These were
synced with path2/dest, which therefore also contains foo and bar.  Now
you rename foo to foo-1 in your source directory.  So we have

$ ls path1/source
foo-1
bar

$ ls path2/dest
foo
bar

When you type path1/source/*, it therefore gets expanded to:
path1/source/foo-1 path1/source/bar

That’s two different source directories, rsync will sync each
separately to a subdirectory of the same name (because there is no
trailing slash) under path2/dest:

path1/source/foo-1 -> path2/dest/foo-1
path1/source/bar   -> path2/dest/bar

And what about path2/dest/foo?  Well, it is not included in these two
syncs, so rsync leaves it untouched.  The --delete option only affects
what is under your sources and targets, here foo-1 and bar, not foo.

So, to conclude, what you probably want is to use one of the rsync
commands I listed above, which sync the entire source directory with the
target, and will clean up anything under the target that is not under
the source.  Instead, what your old command was doing is to look at
every subdirectory and file of the source one by one, which will miss
anything in the target that does not have a corresponding item in the
source.

I hope this helps,

                                                              Sincerely,

                                                                 Bas


--
Sebastiaan L. Zoutendijk • slzoutendijk@gmail.com

Re: [gentoo-user] rsync not deleting removed directory and drive PW DIS 3.3v question

[Dale]

Sebastiaan L. Zoutendijk wrote:
> Dear Dale,
>
> On Friday 2 September 2022, 3.35am -0500, Dale wrote:
>
>> time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Video/*
>> /mnt/10tb/Video/
> What is going on here is due to a subtle interplay of rsync’s syntax and
> shell wildcard expansion.  You can read about the details of rsync
> syntax in its man page, but I’ll illustrate here:
>
> rsync has two ways to specify a source directory.
>
> 1.  Without a trailing slash: `rsync <options> path1/source path2/dest`.
>     This syncs path1/source -> path2/dest/source.
>
> 2.  With a trailing slash: `rsync <options> path1/source/ path2/dest`.
>     This syncs path1/source -> path2/dest.
>
> Note that the two ways lead to different target directories getting
> synced with the source directory.  I think the command you are looking
> for is therefore
>
> time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Video /mnt/10tb
>
> or
>
> time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Video/
> /mnt/10tb/Video
>
> NOTE: I have not tested these commands, I would advise you to run them
> with --dry-run first.
>
> Now, why is your original command not deleting the old subdirectory?
> Let me illustrate this with another example.  Suppose we have the same
> directories as in my first example, path1/source and path2/dest.  Now
> suppose path1/source contains subdirectories foo and bar.  These were
> synced with path2/dest, which therefore also contains foo and bar.  Now
> you rename foo to foo-1 in your source directory.  So we have
>
> $ ls path1/source
> foo-1
> bar
>
> $ ls path2/dest
> foo
> bar
>
> When you type path1/source/*, it therefore gets expanded to:
> path1/source/foo-1 path1/source/bar
>
> That’s two different source directories, rsync will sync each
> separately to a subdirectory of the same name (because there is no
> trailing slash) under path2/dest:
>
> path1/source/foo-1 -> path2/dest/foo-1
> path1/source/bar   -> path2/dest/bar
>
> And what about path2/dest/foo?  Well, it is not included in these two
> syncs, so rsync leaves it untouched.  The --delete option only affects
> what is under your sources and targets, here foo-1 and bar, not foo.
>
> So, to conclude, what you probably want is to use one of the rsync
> commands I listed above, which sync the entire source directory with the
> target, and will clean up anything under the target that is not under
> the source.  Instead, what your old command was doing is to look at
> every subdirectory and file of the source one by one, which will miss
> anything in the target that does not have a corresponding item in the
> source.
>
> I hope this helps,
>
>                                                               Sincerely,
>
>                                                                  Bas
>
>
> --
> Sebastiaan L. Zoutendijk • slzoutendijk@gmail.com
>
>


Replying here for Ramon too. 

When I read Ramon's reply earlier, I remembered that the trailing slash
and even the "*" I had in there was likely the cause.  I recalled
reading how that affects all sorts of things when it comes to copying or
moving files ages ago.  Thing is, it can get a little complicated. 
However, once one understands how it works, it kinda makes sense that it
works that way.  :-D  Reading both replies helped jog my memory, which
is awful nowadays.  :-(

So for anyone running up on either of these two great explanations, this
is what worked for me. 


time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Videos
/mnt/10tb/


When I ran that, it deleted directories that I had changed the names of
or something that were no longer on the source but were still on the
target.  Basically, for the source I removed the "/*" part and the
directory on the target.  It freed up quite a bit of space too.  That
works like I expect it to. 

Now to go fix the other backup script thingys so they work right.  No
telling how many of those have deleted directories on them. 

Thanks much to both.  Reminded me of something I'd forgot about. 
Something we should all remember but sadly, I didn't.  :/

Dale

:-)  :-) 

P. S.  I ordered a 14TB drive. It's a Seagate EXOS X18 14TB
ST14000NM000J which I think doesn't have that pin 3 problem.  I got it
for $150.  My plan, replace one of the 8TB drives in my system, then I
can put the 8TB drive in whatever storage solution I build.  I also
measured my current fire safe, I can't find a case that will fit in
there that holds enough drives.  I'd have to go with a ITX, Raspberry Pi
type thing to get it to fit in the safe and even that could be a tight
squeeze.  I could put it in a out building but I really hate to put
anything electronic in it.  Stuff grows in there, rust too.  Air is
pretty stale and just not good for anything electronic.  This fast
internet is causing some problems but I'm still loving it. :-D

Re: [gentoo-user] rsync not deleting removed directory and drive PW DIS 3.3v question

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

Am Fri, Sep 02, 2022 at 03:35:56AM -0500 schrieb Dale:

> time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Video/*
> /mnt/10tb/Video/

A little OT, but still related:
Instead of -v, give -i a try. -v only shows you the path of synced files. In
addition to that, -i also shows you *why* a file is synced and what exactly is
synced (the whole file, or only permissions or timestamps or other attributes).


An example:

## create source and destination dir, and a source file
$ mkdir a b
$ touch a/foo

## rsync the -v way
$ rsync -av a/ b/
sending incremental file list
./
foo


## update timestamp of first file, create another file, and rsync with -i
$ touch a/{foo,bar}

$ rsync -ai a/ b/
.d..t...... ./
>f+++++++++ bar
>f..t...... foo

See what it did there? Rsync tells you that it copied the new file "bar",
but "foo" only had its timestamp changed, the same as for the root dir,
because we added a file to it.


-- 
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

Three nuns cycle along a country road.
1: “I’ve never come this way before.”
2: “It’s the cobblestones.”

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] rsync not deleting removed directory and drive PW DIS 3.3v question

[Dale]

Frank Steinmetzger wrote:
> Am Fri, Sep 02, 2022 at 03:35:56AM -0500 schrieb Dale:
>
>> time rsync -auv --progress --delete /home/dale/Desktop/Crypt/Video/*
>> /mnt/10tb/Video/
> A little OT, but still related:
> Instead of -v, give -i a try. -v only shows you the path of synced files. In
> addition to that, -i also shows you *why* a file is synced and what exactly is
> synced (the whole file, or only permissions or timestamps or other attributes).
>
>
> An example:
>
> ## create source and destination dir, and a source file
> $ mkdir a b
> $ touch a/foo
>
> ## rsync the -v way
> $ rsync -av a/ b/
> sending incremental file list
> ./
> foo
>
>
> ## update timestamp of first file, create another file, and rsync with -i
> $ touch a/{foo,bar}
>
> $ rsync -ai a/ b/
> .d..t...... ./
>> f+++++++++ bar
>> f..t...... foo
> See what it did there? Rsync tells you that it copied the new file "bar",
> but "foo" only had its timestamp changed, the same as for the root dir,
> because we added a file to it.
>
>

While it may be a tiny bit off topic, not that I mind, it is certainly
interesting.  I added the i to the commands to see what info it
included.  Then I googled for more details on what things meant.  It
shows a d for directory, t for time stamp change etc etc.  While I
rarely use dry-run, it is still neat to see why it is doing something. 
It doesn't take up much room but does provide lots of info.  So, while
it may be of more use if I were using dry-run, I'm still adding it. 

I might add, I got all my so called scripts updated and it is deleting
removed stuff like I expect.  It also hit me why it would add but not
delete.  It would add because there were files inside those
directories.  It had to create one to update the other.  However, if I
removed the directory, well, it wasn't exactly looking since my method
was telling it not too.  ;-)

For anyone running up on this, the same logic of trailing / or * will
likely apply to cp or scp as well. 

I also had a brain storm about my new 14TB drive.  I'm going to make it
my backup drive in place of the 10TB which is almost full already.  I'll
replace a 8TB with the 10TB, which is well tested by now, and it will
add more space.  Later on, maybe another 14TB and remove a 8TB or
something.  As I'm doing this, I'm building up a stock pile of spare
drives, either in the event of a failure or building a nifty new
machine.  o_O 

I just wish I had a better way of tracking passwords for encrypted
drives.  I have a different one for each drive with similar methodology
but no one else would likely guess what they are.  Plus, they are very
secure according to several password checker websites.  Still, good
passwords tend to also be hard to remember.  The more passwords one has,
the harder it gets.  Having a tool sort of defeats the purpose so not
sure of a better way. 

Thanks to all.  Backups are looking great, I just have to do them more
often with this faster internet.  ;-)

Dale

:-)  :-)