[gentoo-user] data recovery advice needed

[gentoo-user] data recovery advice needed

[Jack]

Some months ago, I borked a laptop HDD by trying to move a partition in  
a way that left both the old and new partitions invalid.  (For my  
sanity, I've forced the details out of my memory, but the old and new  
locations overlapped, and I think the move might have been  
interrupted.  My own fault, I know.)  In addition, the drive started  
having read errors, so it was time to replace it anyway.

So, I removed that HDD for safekeeping (completely reinstalled the  
laptop on a new drive) and now I'm trying to recover data from an  
intact partition on the old drive, the problem being that the drive is  
giving some read errors, so I want to minimize access, lest it die  
completely.  I have the drive attached with a SATA to USB adaptor, and  
mounted the partition.  I made the mistake of trying to copy one  
directory (and everything below it) with "cp -r /path/to/mounted/folder  
/path/to/new/location".  As far as I can tell, all the files got copied  
(slowly) but all the new files have the timestamp of when they were  
created.  I know I should have done a "cp -ar" to preserve timestamps.   
Is there any way to fix this other than completely repeating the copy?   
Will some version of rsync do what I want?

Thanks for any suggestions, other than to think more carefully before  
typing.


Jack

Re: [gentoo-user] data recovery advice needed

[Grant Taylor]

On 12/15/18 4:33 PM, Jack wrote:
> Is there any way to fix this other than completely repeating the copy?  

I'm sure there are other ways, but the first things that comes to mind 
is touch.  Or rather a script that uses touch, and possibly find or a 
recursive glob.  The idea being to have the list of files in (either of) 
the directories and touch the new copy with the old copy as the date & 
timestamp reference.

touch --reference=${OldPath}/${CurrentFile} ${NewPath}/${CurrentFile}

I'd probably cd to the new copy's path and do the recursive find / glob 
there to get the file name in a variable.  I think that would apply more 
of the directory traversal walk I/O on the new path and just reference 
the full path to files in the old path.

Quick testing (of globing, namely "**/*") in Zsh makes me think that the 
following would come close.

OldPath="/mnt/old"
NewPath="/mnt/new"
cd $NewPath
for CurrentFile in **/*; do
    touch --reference=${OldPath}/${NewPath} ${NewPath}/${CurrentFile}
done

> Will some version of rsync do what I want?

Maybe ~> probably.

> Thanks for any suggestions, other than to think more carefully before 
> typing.

Mistakes happen.  Once you realize how to recover from something, you 
become less scared of doing it.  Save for your time to actually do the 
recovery.

Re: [gentoo-user] data recovery advice needed

[Rich Freeman]

On Sat, Dec 15, 2018 at 6:33 PM Jack <ostroffjh@users.sourceforge.net> wrote:
>
> So, I removed that HDD for safekeeping (completely reinstalled the
> laptop on a new drive) and now I'm trying to recover data from an
> intact partition on the old drive, the problem being that the drive is
> giving some read errors, so I want to minimize access, lest it die
> completely.

Ok, step 1 - make a copy of the drive before you go messing with it.
I suggest using ddrescue for this.  Basically it works like dd
(creates an image of the drive), but it is persistent in the face of
read errors.

Once you have a copy of the drive you can now start experimenting.
Obviously keep that copy safe and if you want to write to it create
another copy.

As far as fixing dates goes - the touch suggestion might work but
honestly unless you're in a super hurry I'd just do another copy.

If you've lost any kind of drive metadata such that files are missing
completely there are utilities that can scan disk blocks looking for
things like text files, or jpegs.  That is going to be a massive
headache but if you've lost something indispensible it is an option.

Ultimately your goal is going to be to get the files you care about
off the drive.  Then you can just copy/rsync them to a completely
fresh filesystem - I wouldn't go trying to copy the old filesystem
using dd if it has been subject to read errors or especially partial
overwrites of metadata.  You want the metadata to be clean.

And then once you have your data back on a disk give some thought to
your backup strategy.  If you care about data enough to be going
through trouble to rescue it, you should probably have a backup of it.
When I was messing around with btrfs and the filesystem ate my data I
wasn't messing around with hex editors - I just wiped the filesystem
and rsynced from a backup.  Sure, it takes time, but you know the
filesystem is completely clean when you're done.

-- 
Rich

Re: [gentoo-user] data recovery advice needed

[Jack]

Thanks to both Rich and Grant for suggestions.  Part of my issue is  
that I currently have two different failing/failed HDDs to deal with,  
and one of them has two different problems.  At this point, although  
there are files I would really love to recover from each of them, it's  
not critical, so I'm taking my time.  I'll post more details or at  
least status upste later, depending on how things work out.

On 2018.12.15 19:19, Rich Freeman wrote:
> On Sat, Dec 15, 2018 at 6:33 PM Jack  
> <ostroffjh@users.sourceforge.net> wrote:
> >
>> So, I removed that HDD for safekeeping (completely reinstalled the  
>> laptop on a new drive) and now I'm trying to recover data from an  
>> intact partition on the old drive, the problem being that the drive  
>> is giving some read errors, so I want to minimize access, lest it  
>> die completely.
> 
> Ok, step 1 - make a copy of the drive before you go messing with it.  
> I suggest using ddrescue for this.  Basically it works like dd  
> (creates an image of the drive), but it is persistent in the face of  
> read errors.
ddrescue has now been running for almost 22 hours, and it's been 47  
seconds less than that since its last successful read.   The odd thing  
(to me, although it probably should not be) is that the first issue  
with this drive was my own stupidity for trying to move a partition in  
a way I should have known would not work.  (Simplified if inaccurate  
description is a 300GB drive with 100GB free and then a 200GB  
partition, which I tried to move to the beginning of the drive.)  That  
garbled the partition, but the drive itself still worked fine.  Since  
then, the drive itself has started giving lots of read errors, thus the  
slow progress of ddrescue.)  The other drive failed over a shorter  
period of time, even though SMART testing said all was OK.  One of the  
failures led to fsck "fixing"lots of stuff, but truncating or otherwise  
effectively destroying lots of files.  It's a SATA drive, now connected  
to a laptop (only place I have enough recovery space on a good drive)  
by a SAS to USB adaptor.
> 
> Once you have a copy of the drive you can now start experimenting.  
> Obviously keep that copy safe and if you want to write to it create  
> another copy.
understood and agreed.
> 
> As far as fixing dates goes - the touch suggestion might work but  
> honestly unless you're in a super hurry I'd just do another copy.
once the ddrescue works, any approach will work.  However, neither is  
likely to be very helpful with the number of read failures I'm getting.
> 
> If you've lost any kind of drive metadata such that files are missing  
> completely there are utilities that can scan disk blocks looking for  
> things like text files, or jpegs.  That is going to be a massive  
> headache but if you've lost something indispensible it is an option.
I did one run of testdisk on one partition on the drive months ago.   
One problem is that the extension on the recovered files is not  
necessarily the original extension (even if the file is actually  
recoverable) and there seem to be some file types is doesn't (or didn't  
then) know about.  I may well try it again after much more reading on  
tuning its behavior.
> 
> Ultimately your goal is going to be to get the files you care about  
> off the drive.  Then you can just copy/rsync them to a completely  
> fresh filesystem - I wouldn't go trying to copy the old filesystem  
> using dd if it has been subject to read errors or especially partial  
> overwrites of metadata.  You want the metadata to be clean.
right.
> 
> And then once you have your data back on a disk give some thought to  
> your backup strategy.  If you care about data enough to be going  
> through trouble to rescue it, you should probably have a backup of  
> it. When I was messing around with btrfs and the filesystem ate my  
> data I wasn't messing around with hex editors - I just wiped the  
> filesystem and rsynced from a backup.  Sure, it takes time, but you  
> know the filesystem is completely clean when you're done.
Someone (on this list?) had a signature "There are two kinds of people:  
those who do backups and those who have never had a hard drive fail."   
I guess there's actually a third.....
> 
> --
> Rich
Jack

Re: [gentoo-user] data recovery advice needed

[Rich Freeman]

On Mon, Dec 17, 2018 at 2:32 PM Jack <ostroffjh@users.sourceforge.net> wrote:
>
> ddrescue has now been running for almost 22 hours, and it's been 47
> seconds less than that since its last successful read.

There is of course no guarantee that it will EVER successfully read
all your data.  You might be able to tell it to skip blocks and move
on.  Obviously all it can do is keep asking the drive to try again.
If you want better than that then you're talking clean room rescue
measures.

> I did one run of testdisk on one partition on the drive months ago.
> One problem is that the extension on the recovered files is not
> necessarily the original extension (even if the file is actually
> recoverable) and there seem to be some file types is doesn't (or didn't
> then) know about.  I may well try it again after much more reading on
> tuning its behavior.

Even in the best of circumstances this sort of approach is
painstaking.  If that drive has the only photos of your kid's birth it
might be worth the trouble to you.  If you're trying to rescue your
bittorrent collection I'd suggest moving on.

Or you can try to pay somebody to do it for you.  There is a reason
those recovery companies charge an arm and a leg.  People who are
desperate enough to need their services don't have many other
options...

-- 
Rich

Re: [gentoo-user] data recovery advice needed

[Grant Taylor]

On 12/17/2018 12:42 PM, Rich Freeman wrote:
> There is of course no guarantee that it will EVER successfully read all 
> your data.  You might be able to tell it to skip blocks and move on. 
> Obviously all it can do is keep asking the drive to try again.  If you 
> want better than that then you're talking clean room rescue measures.

As long as the drive isn't going into physical failure; bearings, 
thermal death, etc, you may be okay.

I'm a fan of SpinRite.  I've had SpinRite recover things that everything 
else said was impossible to recover.  SpinRite addresses media issues 
and helps the drive detect that there is a problem.  SpinRite also has a 
mode, DynaStat, that it will try many many many times to read a given 
sector using all sorts of tricks, head seek patterns, etc.  (You can 
also alter it's default behavior to try more or fewer times.)

There is no trial copy of SpinRite.  But GRC does have a good return 
policy.  If you buy SpinRite to recover your drive, and are unsatisfied, 
they will very very extremely likely give you your money back.  (There 
is a tiny fraction of a chance that they won't refund your money if you 
don't try the things they ask to make sure that you aren't using 
SpinRite properly.)  So, it's not like you are out money if it doesn't 
work.  (I think it's MSRP of ~$90.)

Aside:  It sounds like your drive might think that it has problems 
reading the media, which is SpinRite's domain.

Make sure the drive is not overheating.  Even a lowly 80 mm case fan 
will move more than enough air to keep the average drive cool enough, 
even during heavy use / data recovery.

Once you know that you have good magnetic media, then you can start 
playing to reconstruct your partition.

So, SpinRite is an option before places like DriveSavers (or what ever 
name they go by).  Even they have a no-guarantee of recovery policy.

> If you're trying to rescue your bittorrent collection I'd suggest 
> moving on.

I misread "bittorrent" as "bitcoin".  That /might/ be another story. 
Depending on how many BTC you have.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] data recovery advice needed

[Grant Taylor]

On 12/17/2018 12:32 PM, Jack wrote:
> The other drive failed over a shorter period of time, even though SMART 
> testing said all was OK.

IMHO, by the time that S.M.A.R.T. complains, most chances of data 
retrieval are gone.

I treat S.M.A.R.T. errors as a confirmation that the drive realizes that 
there is an imminent or past tense catastrophic failure.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] data recovery advice needed

[Wols Lists]

On 17/12/18 19:32, Jack wrote:
> ddrescue has now been running for almost 22 hours, and it's been 47
> seconds less than that since its last successful read. 

Doesn't sound good. Just to throw my tuppence-worth of bad news into the
mix, are your drives in USB enclosures? I really don't trust USB and
hard drives, I'd guess maybe 80% of my drive failures have been that
combination :-(

I've currently got two drives, in USB caddies, that I want to try and
salvage, and at some point soon I'll be buying an external eSATA adaptor
- that does NOT enclose the drives to give free airflow.

The other thing is, if you're having trouble reading the drive, could it
be the USB bus that's playing up? Again, I'm very wary of large amounts
of data over USB, it just doesn't seem to work for me.

Cheers,
Wol

Re: [gentoo-user] data recovery advice needed

[Dale]

Wols Lists wrote:
> On 17/12/18 19:32, Jack wrote:
>> ddrescue has now been running for almost 22 hours, and it's been 47
>> seconds less than that since its last successful read. 
> Doesn't sound good. Just to throw my tuppence-worth of bad news into the
> mix, are your drives in USB enclosures? I really don't trust USB and
> hard drives, I'd guess maybe 80% of my drive failures have been that
> combination :-(
>
> I've currently got two drives, in USB caddies, that I want to try and
> salvage, and at some point soon I'll be buying an external eSATA adaptor
> - that does NOT enclose the drives to give free airflow.
>
> The other thing is, if you're having trouble reading the drive, could it
> be the USB bus that's playing up? Again, I'm very wary of large amounts
> of data over USB, it just doesn't seem to work for me.
>
> Cheers,
> Wol
>
>


I have hit and miss issues with USB at times as well. I ordered a sata
external enclosure that has a fan for cooling as well.  Some external
enclosures have both sata and USB ports.  That way you can use sata when
possible. 

I ordered three USB enclosures and only one works well.  The others
either don't work at all or are very slow. 

I might add, I've read where people who make their own external drives,
buy the drive and enclosure separately then assemble it themselves, put
the drive into their computer to format with a file system and then put
it in the enclosure for normal use.  Some think it formats better hooked
directly to a sata port inside the puter than it does over USB.  I'm not
so sure myself.  In my experience, if it works well, it works well all
the way around. 

May buy me a card that has both internal and external ports later on. 
Sale maybe??

Dale

:-)  :-) 

Re: [gentoo-user] data recovery advice needed

[Grant Taylor]

On 12/18/2018 05:21 AM, Wols Lists wrote:
> I really don't trust USB and hard drives

IMHO external USB drive enclosures (physical enclosure, USB to SATA / 
PATA / ??? adapter, (no)fan) are suboptimal at best and are most likely 
to work when all the components are happy.

I view them as the weakest link.

If I have a drive problem, I like to connect the drive to a SATA (PATA) 
connector and access the drive directly.  I usually have better results 
saving data that way.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] data recovery advice needed

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

Am Dienstag, 18. Dezember 2018, 16:56:02 CET schrieb Grant Taylor:
> On 12/18/2018 05:21 AM, Wols Lists wrote:
> > I really don't trust USB and hard drives
> 
> IMHO external USB drive enclosures (physical enclosure, USB to SATA /
> PATA / ??? adapter, (no)fan) are suboptimal at best and are most likely
> to work when all the components are happy.
> 
> I view them as the weakest link.
> 
> If I have a drive problem, I like to connect the drive to a SATA (PATA)
> connector and access the drive directly.  I usually have better results
> saving data that way.

Yeah, that matches my experience as well.

Just to add a personal anecdote: I once had an external drive that I thought 
had broken (it made the "clicks of death"), but after taking it out of its USB 
enclosure and connecting it directly via IDE (shows how old the drive was...) 
it actually worked and I was able to copy all data from it.  Only one 
(unimportant) directory had been corrupted.

I mean, I still use USB drives, as they're quite practical for some things, 
but I'm more aware of the potential issues with them now.

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] data recovery advice needed

[Mick]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

On Tuesday, 18 December 2018 21:28:38 GMT Marc Joliet wrote:

> Just to add a personal anecdote: I once had an external drive that I thought
> had broken (it made the "clicks of death"), but after taking it out of its
> USB enclosure and connecting it directly via IDE (shows how old the drive
> was...) it actually worked and I was able to copy all data from it.  Only
> one (unimportant) directory had been corrupted.

I had a SATA drive in a desktop which suddenly started clicking rather noisily 
on start up.  I thought it's time was up, but I was advised in this M/L to 
reseat the SATA connector on the MoBo.  The drive behaved thereafter without 
any more clicking noises.  Over the years I had to repeat this reseating of 
the connector a couple of times, but otherwise it has given me faultless 
service.  I'm not saying Mark's ATA-to-USB was not to blame, but faulty 
connectors can cause such clicking symptoms.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] data recovery advice needed

[Jack]

On 2018.12.18 07:21, Wols Lists wrote:
> On 17/12/18 19:32, Jack wrote:
>> ddrescue has now been running for almost 22 hours, and it's been 47  
>> seconds less than that since its last successful read.
> 
> Doesn't sound good. Just to throw my tuppence-worth of bad news into  
> the mix, are your drives in USB enclosures? I really don't trust USB  
> and hard drives, I'd guess maybe 80% of my drive failures have been  
> that combination :-(
No enclosures involved at all.  In one case the 2.5" drive was in the  
laptop when the partition move issue occurred.  It has been just  
sitting in the open when the read errors started - initially connected  
by SATA to the desktop with the side open, and now by USB connected to  
the laptop.  (I'll move back to the desktop once I get a new disk with  
enough room to do the recovery there.)  The other drive (3.5") was in a  
desktop when it failed, and other than a brief attempt at using the USB  
connection (no enclosure) has just been sitting inside one or the other  
desktop.  Same issue needing more space to attempt any further recovery.
> 
> I've currently got two drives, in USB caddies, that I want to try and  
> salvage, and at some point soon I'll be buying an external eSATA  
> adaptor - that does NOT enclose the drives to give free airflow.
> 
> The other thing is, if you're having trouble reading the drive, could  
> it be the USB bus that's playing up? Again, I'm very wary of large  
> amounts of data over USB, it just doesn't seem to work for me.
> 
> Cheers,
> Wol
Jack

Re: [gentoo-user] data recovery advice needed

[Heiko Baums]

Am Sat, 15 Dec 2018 18:33:35 -0500
schrieb Jack <ostroffjh@users.sourceforge.net>:

> Some months ago, I borked a laptop HDD by trying to move a partition
> in a way that left both the old and new partitions invalid.  (For my  
> sanity, I've forced the details out of my memory, but the old and
> new locations overlapped, and I think the move might have been  
> interrupted.  My own fault, I know.)

Have you written anything on this HDD after moving those partitions?
And did you move the partitions or did you repartition the HDD?

You could try to find the beginning of the old partitions with hexedit
and set a pointer to this position with losetup. Set the offset to this
position. Linux actually doesn't need partitions to access the HDD
and/or mount file systems. Partitions just make it a lot easier.

And keep in mind that partitions are written to the partition table at
the beginning of the drive. Repartitioning usually doesn't overwrite
any data. So if you haven't written anything onto the new partitions
(incl. formatting the new partitions) all the data is most likely still
there.

So you could also try to recover your old partition table. Maybe the
backup of your old super block is still intact. Unfortunately I can't
remember how to find it out. Maybe with fdisk resp. gdisk or with
testdisk.

But there is a chance that the behavior of moving partitions is a bit
different than repartitioning the HDD, so that the data is actually
already overwritten by moving those partitions.

On the other hand, if you had really moved the partitions the
partitioning tool you used should have seen that both partitions
would overlap and shouldn't have done this.

That said, no matter which problem you have with an HDD, always stay
calm and think first before you do anything. In most cases you can
recover your data. But the first step is to stop working (particularly
writing) with this HDD. The second step is - as the others already
mentioned - to make a copy with dd or ddrescue.

I had a lot of HDD crashes, logical and physical ones. And once I even
didn't have a backup. I never lost any data so far, not even a bit.

Heiko

Re: [gentoo-user] data recovery advice needed

[Jack]

On 2018.12.17 17:32, Heiko Baums wrote:
> Am Sat, 15 Dec 2018 18:33:35 -0500
> schrieb Jack <ostroffjh@users.sourceforge.net>:
> 
>> Some months ago, I borked a laptop HDD by trying to move a partition  
>> in a way that left both the old and new partitions invalid.  (For my  
>> sanity, I've forced the details out of my memory, but the old and  
>> new locations overlapped, and I think the move might have been  
>> interrupted.  My own fault, I know.)
> 
> Have you written anything on this HDD after moving those partitions?  
> And did you move the partitions or did you repartition the HDD?
At this point, unfortunately, my memory is hazy.  I did not  
repartition, but the damage made accessing anything difficult (see  
details below).  The additional problem is that now I'm getting  
physical read errors on the drive, so to be safe, I'd need to get  
another drive large enough to use ddrescue, and then go digging.
> 
> You could try to find the beginning of the old partitions with  
> hexedit and set a pointer to this position with losetup. Set the  
> offset to this position. Linux actually doesn't need partitions to  
> access the HDD and/or mount file systems. Partitions just make it a  
> lot easier.
At this point, I think I've forgotten the details, but using the  
example of a 300G drive with 100G empty and then a 200G partition, when  
I moved the partition to the beginning of the disk (using gparted, as I  
remember) once it moved more than the first 100G of the partition, it  
overwrote the beginning of the original partition, and once it  
overwrote any of the directory structure it still needed to know where  
stuff was, game over.   The first 100G of the "new" partition is  
essentially intact, but the damage to the beginning of the "old"  
partition makes it difficult to start.  I suppose I could go looking  
for any intact directory inodes and see whether it's intact enough to  
find anything of interest.  (There are really only a few files I would  
still like to recover - but as they are in fact .VDI files, it's likely  
all or nothing, because I don't think just part of one of them will be  
at all useful.)  In fact, it would be more acurate to say there was  
another 100G before all of that, and that partition is still intact,  
except for the read errors.
> 
> And keep in mind that partitions are written to the partition table  
> at the beginning of the drive. Repartitioning usually doesn't  
> overwrite any data. So if you haven't written anything onto the new  
> partitions (incl. formatting the new partitions) all the data is most  
> likely still there.
The "new" partition is incomplete, but essentially intact, as much of  
it is there.  The beginning of the "old" partition is overwitten with  
stuff from somewhere in the middle of the partition being moved.
> 
> So you could also try to recover your old partition table. Maybe the  
> backup of your old super block is still intact. Unfortunately I can't  
> remember how to find it out. Maybe with fdisk resp. gdisk or with  
> testdisk.
I know the location of all the partition starts and ends, so that's not  
the issue.  I just need lots more empty space to put the copies so I  
can safely go digging.
> 
> But there is a chance that the behavior of moving partitions is a bit  
> different than repartitioning the HDD, so that the data is actually  
> already overwritten by moving those partitions.
Exactly.
> 
> On the other hand, if you had really moved the partitions the  
> partitioning tool you used should have seen that both partitions  
> would overlap and shouldn't have done this.
I think I mentioned that.  I'm going to have to do some testing, and  
probably learn that gparted is smarter than I thought, which means I've  
clearly forgotten something else stupid that I did.  Otherwise, I'll  
file a bug against gparted.
> 
> That said, no matter which problem you have with an HDD, always stay  
> calm and think first before you do anything. In most cases you can  
> recover your data. But the first step is to stop working  
> (particularly writing) with this HDD. The second step is - as the  
> others already mentioned - to make a copy with dd or ddrescue.
All that would be good and "easy" (for some definition) if the disk  
itself were not in process of failing.
> 
> I had a lot of HDD crashes, logical and physical ones. And once I  
> even didn't have a backup. I never lost any data so far, not even a  
> bit.
I'd call that "famous last words" :-)
> 
> Heiko
Jack

Re: [gentoo-user] data recovery advice needed

[Peter Humphrey]

On Monday, 17 December 2018 23:19:39 GMT Jack wrote:

> At this point, I think I've forgotten the details, but using the
> example of a 300G drive with 100G empty and then a 200G partition, when
> I moved the partition to the beginning of the disk (using gparted, as I
> remember) once it moved more than the first 100G of the partition, it
> overwrote the beginning of the original partition, and once it
> overwrote any of the directory structure it still needed to know where
> stuff was, game over.

Gparted can handle that without difficulty - I do it often - so I think you 
must be mistaken in the tool you used.

Sorry not to be more helpful...

-- 
Regards,
Peter.

Re: [gentoo-user] data recovery advice needed

[Jack]

On 2018.12.18 04:43, Peter Humphrey wrote:
> On Monday, 17 December 2018 23:19:39 GMT Jack wrote:
> 
>> At this point, I think I've forgotten the details, but using the  
>> example of a 300G drive with 100G empty and then a 200G partition,  
>> when I moved the partition to the beginning of the disk (using  
>> gparted, as I remember) once it moved more than the first 100G of  
>> the partition, it overwrote the beginning of the original partition,  
>> and once it overwrote any of the directory structure it still needed  
>> to know where stuff was, game over.
> 
> Gparted can handle that without difficulty - I do it often - so I  
> think you must be mistaken in the tool you used.
No, not mistaken in which tool, but likely mistaken in my memory of the  
exact course of events.  It's possible the gparted move operation was  
interrupted.  Could have been an accidental Ctl-C or a power failure.   
However, at this point, I'd need a better crystal ball looking  
backwards to know for sure.
> 
> Sorry not to be more helpful...
I have more responses for elsewhere in the thread, but I think any  
further serious attempts at recovery are going to have to wait until I  
buy a new disk or two, so I can do everything internal on the desktop,  
and not on the laptop with USB.
> --
> Regards,
> Peter.
Jack

Re: [gentoo-user] data recovery advice needed

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2611 bytes --]

On Tuesday, 18 December 2018 17:11:28 GMT Jack wrote:
> On 2018.12.18 04:43, Peter Humphrey wrote:
> > On Monday, 17 December 2018 23:19:39 GMT Jack wrote:
> >> At this point, I think I've forgotten the details, but using the
> >> example of a 300G drive with 100G empty and then a 200G partition,
> >> when I moved the partition to the beginning of the disk (using
> >> gparted, as I remember) once it moved more than the first 100G of
> >> the partition, it overwrote the beginning of the original partition,
> >> and once it overwrote any of the directory structure it still needed
> >> to know where stuff was, game over.
> > 
> > Gparted can handle that without difficulty - I do it often - so I
> > think you must be mistaken in the tool you used.
> 
> No, not mistaken in which tool, but likely mistaken in my memory of the
> exact course of events.  It's possible the gparted move operation was
> interrupted.  Could have been an accidental Ctl-C or a power failure.
> However, at this point, I'd need a better crystal ball looking
> backwards to know for sure.
> 
> > Sorry not to be more helpful...
> 
> I have more responses for elsewhere in the thread, but I think any
> further serious attempts at recovery are going to have to wait until I
> buy a new disk or two, so I can do everything internal on the desktop,
> and not on the laptop with USB.
> 
> > --
> > Regards,
> > Peter.
> 
> Jack

I know others have commented on the reliability of recovering data from drives 
connected via USB caddy, but I have had satisfactory results on a number of 
cases.

The last time I tried to recover data from a failing disk, which had SMART 
End-to-End errors increasing continuously and read errors with ddrescue going 
up each time I ran it.

I cloned the whole drive having run ddrescue backwards and forwards a couple 
of times.  c/f/gdisk would see all partitions, but when I tried to mount the 
cloned /dev/sdb4 (NTFS) with ntfs-3g it complained there was no device found 
(/dev/sdb4).  I got the same error with the failing drive.

So, I used losetup with --offset on the failing drive itself over USB 2.0 and 
was able to mount and recover all the NTFS files.

Over the years I've used clonezilla, ddrescue, testdisk, photorec and losetup 
to recover files.  On a couple of times where data on the disk had been 
overwritten by subsequent operations, I was not able to recover the affected 
files.  So, if when moving the partition data was overwritten I suspect it 
will be very difficult to recover this with conventional software tools.  
However, it doesn't hurt to try.  :-)

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] data recovery advice needed

[Jack]

On 2018.12.18 12:42, Mick wrote:
[snip...]
> So, I used losetup with --offset on the failing drive itself over USB  
> 2.0 and was able to mount and recover all the NTFS files.
I definitely need to read up on that one - I totally unfamiliar with it.
> 
> Over the years I've used clonezilla, ddrescue, testdisk, photorec and  
> losetup to recover files.  On a couple of times where data on the  
> disk had been overwritten by subsequent operations, I was not able to  
> recover the affected files.  So, if when moving the partition data  
> was overwritten I suspect it will be very difficult to recover this  
> with conventional software tools. However, it doesn't hurt to try.   
> :-)
I should be in good shape there.  The partition's new location should  
have the first half intact, and since the overwriting was of the first  
part of the old location, it's second half should be intact.  The files  
should all be there - but I imagine I might have to play with any  
directories which were successfully move, but not the stuff they point  
to.  Of course, all of this depends on recovering from the read errors  
- so ddrescue needs to finish first.
> 
> --
> Regards,
> Mick
Jack

Re: [gentoo-user] data recovery advice needed

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2630 bytes --]

On Tuesday, 18 December 2018 17:49:37 GMT Jack wrote:
> On 2018.12.18 12:42, Mick wrote:
> [snip...]
> 
> > So, I used losetup with --offset on the failing drive itself over USB
> > 2.0 and was able to mount and recover all the NTFS files.
> 
> I definitely need to read up on that one - I totally unfamiliar with it.

If you are not able to mount a partition because of a 'device not found ...' 
type of error use c/f/gdisk et al. to find the sector at which the partition 
in question starts.  Let's say gdisk shows partition 6 starts at sector 
1,615,872 and the sector size is 512 bytes.  The starting offset for losetup 
should be:

1,615,872 x 512 = 827,326,464

Similarly, let's say the size of the partition is shown as being 1,910,121,163 
sectors, which gives a size of 977,982,035,456 in bytes:

1,910,121,163 x 512 = 977,982,035,456


Then add the above parameters in the losetup command to create a loopback 
device as follows:

losetup -o 827326464 --sizelimit 977982035456 --sector-size 512 -P -r -v /dev/
loop4 /dev/sdb

and subsequently try to mount it:

mkdir /mnt/recovery

mount /dev/loop4 /mnt/recovery


> > Over the years I've used clonezilla, ddrescue, testdisk, photorec and
> > losetup to recover files.  On a couple of times where data on the
> > disk had been overwritten by subsequent operations, I was not able to
> > recover the affected files.  So, if when moving the partition data
> > was overwritten I suspect it will be very difficult to recover this
> > with conventional software tools. However, it doesn't hurt to try.
> > 
> > :-)
> 
> I should be in good shape there.  The partition's new location should
> have the first half intact, and since the overwriting was of the first
> part of the old location, it's second half should be intact.  The files
> should all be there - but I imagine I might have to play with any
> directories which were successfully move, but not the stuff they point
> to.  Of course, all of this depends on recovering from the read errors
> - so ddrescue needs to finish first.

This is what I would try:

1. dd the previously moved first half to a new partition/device.

2. dd the intact second half from the corresponding second half offset 
position of the overwritten partition to the new partition/device.  Use --seek 
and --skip in the dd command as appropriate to achieve this.

This way you should be able to stitch together the two halves, hoping that no 
bits/bytes were lost when the move/overwritting occurred back then.  If the 
original filesystem is not recoverable you should be able to recover 
individual files with photorec.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] data recovery advice needed

[Grant Taylor]

On 12/18/2018 10:49 AM, Jack wrote:
> I should be in good shape there.  The partition's new location should 
> have the first half intact, and since the overwriting was of the first 
> part of the old location, it's second half should be intact.  The files 
> should all be there - but I imagine I might have to play with any 
> directories which were successfully move, but not the stuff they point 
> to.  Of course, all of this depends on recovering from the read errors - 
> so ddrescue needs to finish first.

I feel like you have two ~½ partitions that you need to aggregate and 
try to access.  I half way expect that you need to image the drive, and 
then extract the two ½s and put them back together.

I don't know if losetup can ignore something in the middle or not.

I would then expect that your file system in the new spliced back 
together ""partition to be okay.

I /think/ that most modern file systems have addressing within the 
partition and not subject to physical location on the disk.  As such, 
you should be able to mount it fairly cleanly.  I would expect that 
files are no more corrupted than they were before the move.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] data recovery advice needed

[Grant Taylor]

On 12/18/2018 10:42 AM, Mick wrote:
> I know others have commented on the reliability of recovering data from drives
> connected via USB caddy, but I have had satisfactory results on a number of
> cases.

I think it completely depends on the type of problem.  In my experience, 
SATA-to-USB adapters don't deal well with physical hard drive / media 
errors.  (At least compared to SATA connections on the motherboard.)  I 
think their retry mechanisms are somewhat limited.  Conversely, software 
/ file system / logical corruption issues are likely perfectly fine over 
USB.

> I cloned the whole drive having run ddrescue backwards and forwards a couple
> of times.  c/f/gdisk would see all partitions, but when I tried to mount the
> cloned /dev/sdb4 (NTFS) with ntfs-3g it complained there was no device found
> (/dev/sdb4).  I got the same error with the failing drive.

Seeing the partitions in the partition table is independent of the 
device file being there.  -  Did the device file exist?  -  I 
occasionally have to run (k)part(x) to tell the kernel that the 
partition is there and to create the device file.

> So, I used losetup with --offset on the failing drive itself over USB 2.0 and
> was able to mount and recover all the NTFS files.

That sounds like the kernel didn't know about the partition and / or the 
device file didn't exist.  losetup will create a new device file (and 
possibly partitions depending on how you did it).  Then you will be able 
to point standard FS tools at the new device (partition) file.

> Over the years I've used clonezilla, ddrescue, testdisk, photorec and losetup
> to recover files.  On a couple of times where data on the disk had been
> overwritten by subsequent operations, I was not able to recover the affected
> files.  So, if when moving the partition data was overwritten I suspect it
> will be very difficult to recover this with conventional software tools.

I would expect moving a partition to an earlier location on the drive 
will take the first block and move it to the new location, then the next 
block, etc. until finished.  I'd expect moving a partition to a later 
location on the drive will take the last block and move it to the next 
location, then the previous block, etc. until finished.  This way the 
data is never actually overwritten.  Blocks have been moved, and 
committed to disk, thus vacating the old block.

> However, it doesn't hurt to try.  :-)

I mostly agree.  You need to be mindful of physical damage and thermals.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] data recovery advice needed

[Mick]

[-- Attachment #1: Type: text/plain, Size: 3014 bytes --]

On Tuesday, 18 December 2018 18:51:55 GMT Grant Taylor wrote:
> On 12/18/2018 10:42 AM, Mick wrote:
> > I know others have commented on the reliability of recovering data from
> > drives connected via USB caddy, but I have had satisfactory results on a
> > number of cases.
> 
> I think it completely depends on the type of problem.  In my experience,
> SATA-to-USB adapters don't deal well with physical hard drive / media
> errors.  (At least compared to SATA connections on the motherboard.)  I
> think their retry mechanisms are somewhat limited.  Conversely, software
> / file system / logical corruption issues are likely perfectly fine over
> USB.

I ran ddrescue while the drive was still on the laptop.  The clone was on the 
USB caddy.


> > I cloned the whole drive having run ddrescue backwards and forwards a
> > couple of times.  c/f/gdisk would see all partitions, but when I tried to
> > mount the cloned /dev/sdb4 (NTFS) with ntfs-3g it complained there was no
> > device found (/dev/sdb4).  I got the same error with the failing drive.
> 
> Seeing the partitions in the partition table is independent of the
> device file being there.  -  Did the device file exist?  -  I
> occasionally have to run (k)part(x) to tell the kernel that the
> partition is there and to create the device file.

The disk block device is there and so is the first partition (only):

ls -la /dev/sdb*
brw-rw---- 1 root disk 8, 16 Dec 19 11:20 /dev/sdb
brw-rw---- 1 root disk 8, 17 Dec 19 11:20 /dev/sdb1


However, there's 6 partitions in total:

Disk /dev/sdb: 1953525168 sectors, 931.5 GiB
Model: LucidPort USB300
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 1A95F5D1-5630-4E06-9DC3-36841C786DDF
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 4770 sectors (2.3 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          821247   400.0 MiB   2700  Basic data partition
   2          821248         1353727   260.0 MiB   EF00  EFI system partition
   3         1353728         1615871   128.0 MiB   0C01  Microsoft reserved 
...
   4         1615872      1911737034   910.8 GiB   0700  Basic data partition
   5      1911738368      1915412479   1.8 GiB     2700  
   6      1915412480      1953523711   18.2 GiB    0700  Basic data partition

I can see partition 4 I was trying to recover, but could not add it:

partx --show --nr 4 /dev/sdb
NR   START        END    SECTORS   SIZE NAME                 UUID
 4 1615872 1911737034 1910121163 910.8G Basic data partition fea85fb3-
cfdb-4868-a1ad-bab264dad237

partx --add --nr 4 /dev/sdb
partx: /dev/sdb: error adding partition 4

partx --add /dev/sdb
partx: /dev/sdb: error adding partitions 1-6

In any case, losetup with offset/size succeeded in mounting it and I was able 
to access the fs on it.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] data recovery advice needed

[Grant Taylor]

On 12/19/2018 04:43 AM, Mick wrote:
> I ran ddrescue while the drive was still on the laptop.  The clone was on the
> USB caddy.

ACK

> The disk block device is there and so is the first partition (only):
> 
> ls -la /dev/sdb*
> brw-rw---- 1 root disk 8, 16 Dec 19 11:20 /dev/sdb
> brw-rw---- 1 root disk 8, 17 Dec 19 11:20 /dev/sdb1
> 
> However, there's 6 partitions in total:
> 
> Disk /dev/sdb: 1953525168 sectors, 931.5 GiB
> Model: LucidPort USB300
> Sector size (logical/physical): 512/512 bytes
> Disk identifier (GUID): 1A95F5D1-5630-4E06-9DC3-36841C786DDF
> Partition table holds up to 128 entries

128 entries tells me that the disk has GPT partition table, not a 
classis MS-DOS / PC-BIOS partition table.

This sounds like the (what I understand to be) the classic protection 
partition that GPT fakes in PC-BIOS partition tables.

Do you have GPT partition support in your kernel?

I'm guessing the one partition that you see is the size of the entire drive.

> Main partition table begins at sector 2 and ends at sector 33
> First usable sector is 34, last usable sector is 1953525134
> Partitions will be aligned on 2048-sector boundaries
> Total free space is 4770 sectors (2.3 MiB)
> 
> Number  Start (sector)    End (sector)  Size       Code  Name
>     1            2048          821247   400.0 MiB   2700  Basic data partition
>     2          821248         1353727   260.0 MiB   EF00  EFI system partition
>     3         1353728         1615871   128.0 MiB   0C01  Microsoft reserved
> ...
>     4         1615872      1911737034   910.8 GiB   0700  Basic data partition
>     5      1911738368      1915412479   1.8 GiB     2700
>     6      1915412480      1953523711   18.2 GiB    0700  Basic data partition
> 
> I can see partition 4 I was trying to recover, but could not add it:
> 
> partx --show --nr 4 /dev/sdb
> NR   START        END    SECTORS   SIZE NAME                 UUID
>   4 1615872 1911737034 1910121163 910.8G Basic data partition fea85fb3-
> cfdb-4868-a1ad-bab264dad237
> 
> partx --add --nr 4 /dev/sdb
> partx: /dev/sdb: error adding partition 4
> 
> partx --add /dev/sdb
> partx: /dev/sdb: error adding partitions 1-6

That really sounds like your kernel doesn't have GPT support (loaded).

> In any case, losetup with offset/size succeeded in mounting it and I was able
> to access the fs on it.

Good.



-- 
Grant. . . .
unix || die

Re: [gentoo-user] data recovery advice needed

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2218 bytes --]

On Wednesday, 19 December 2018 18:46:40 GMT Grant Taylor wrote:
> On 12/19/2018 04:43 AM, Mick wrote:

> > Partition table holds up to 128 entries
> 
> 128 entries tells me that the disk has GPT partition table, not a
> classis MS-DOS / PC-BIOS partition table.
> 
> This sounds like the (what I understand to be) the classic protection
> partition that GPT fakes in PC-BIOS partition tables.
> 
> Do you have GPT partition support in your kernel?
> 
> I'm guessing the one partition that you see is the size of the entire drive.
> > Main partition table begins at sector 2 and ends at sector 33
> > First usable sector is 34, last usable sector is 1953525134
> > Partitions will be aligned on 2048-sector boundaries
> > Total free space is 4770 sectors (2.3 MiB)
> > 
> > Number  Start (sector)    End (sector)  Size       Code  Name
> > 
> >     1            2048          821247   400.0 MiB   2700  Basic data
> >     partition
> >     2          821248         1353727   260.0 MiB   EF00  EFI system
> >     partition
> >     3         1353728         1615871   128.0 MiB   0C01  Microsoft
> >     reserved
> > 
> > ...
> > 
> >     4         1615872      1911737034   910.8 GiB   0700  Basic data
> >     partition
> >     5      1911738368      1915412479   1.8 GiB     2700
> >     6      1915412480      1953523711   18.2 GiB    0700  Basic data
> >     partition
> > 
> > I can see partition 4 I was trying to recover, but could not add it:
> > 
> > partx --show --nr 4 /dev/sdb
> > NR   START        END    SECTORS   SIZE NAME                 UUID
> > 
> >   4 1615872 1911737034 1910121163 910.8G Basic data partition fea85fb3-
> > 
> > cfdb-4868-a1ad-bab264dad237
> > 
> > partx --add --nr 4 /dev/sdb
> > partx: /dev/sdb: error adding partition 4
> > 
> > partx --add /dev/sdb
> > partx: /dev/sdb: error adding partitions 1-6
> 
> That really sounds like your kernel doesn't have GPT support (loaded).

Grant, you're spot on!

Symbol: EFI_PARTITION [=n]

I was under the impression I had it enabled, but clearly I hadn't on this PC; 
which has a legacy BIOS with an MBR disk and which I used for troubleshooting 
the faulty drive after I removed it from the laptop.  Lesson learned.  ;-)

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] data recovery advice needed

[Grant Taylor]

On 12/19/2018 03:16 PM, Mick wrote:
> Grant, you're spot on!

/me looks around wondering what he did and if he needs to run and hide.

> Symbol: EFI_PARTITION [=n]

Oops.  That will certainly mess with you.

> I was under the impression I had it enabled, but clearly I hadn't on 
> this PC; which has a legacy BIOS with an MBR disk and which I used for 
> troubleshooting the faulty drive after I removed it from the laptop. 
> Lesson learned.  ;-)

I've learned the hard way to "trust /but/ *verify*" things.  Even things 
that I think I've done.



-- 
Grant. . . .
unix || die