[gentoo-user] testing a corrupt SD card

[gentoo-user] testing a corrupt SD card

[Iain Buchanan]

Hi all,

recently my SD card just went bonkers.  Unfortunately I lost a lot of
photos on it (backups are useless until the data actually gets to the
backup...) but fortunately I was able to use a program to recover about
170 photos.

Anyway, I don't know if it was just static, shock, dead card, or phase
of the moon, so I would like to see if the card is good before I
continue to use it.

I've reformatted it and I get:
$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mmcblk0p1          500960    500960         0 100% /media/PICS

so I created a file:
dd if=/dev/urandom of=Desktop/random.img bs=1024 count=500960

then copied it to the card, and then copied it back as random-2.img.  If
I md5sum the two files, they are identical:
$ md5sum random*
9dcac25cfd8585be5939c0ff969de310  random-2.img
9dcac25cfd8585be5939c0ff969de310  random.img

Does that mean my memory card is good to go, or should I use some other
method of bad sector detection?

It's a Lexar Media 512Mb SD card, a couple of years old.  Yes I know I
can get a cheap 2Gb for <$20 but I'm more interested in the principle of
the test :)

thanks for any tips!
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

This sentence contradicts itself -- no actually it doesn't.
		-- Douglas Hofstadter

Re: [gentoo-user] testing a corrupt SD card

[Stroller]

On 6 Feb 2009, at 05:28, Iain Buchanan wrote:
> It's a Lexar Media 512Mb SD card, a couple of years old.  Yes I know I
> can get a cheap 2Gb for <$20 but I'm more interested in the  
> principle of
> the test :)

I thought you could get then for < $5, but anyway....

> so I created a file:
> dd if=/dev/urandom of=Desktop/random.img bs=1024 count=500960
>
> then copied it to the card, and then copied it back as  
> random-2.img.  If
> I md5sum the two files, they are identical:
> $ md5sum random*
> 9dcac25cfd8585be5939c0ff969de310  random-2.img
> 9dcac25cfd8585be5939c0ff969de310  random.img
>
> Does that mean my memory card is good to go, or should I use some  
> other
> method of bad sector detection?

I'd be more or less happy with that methodology, had I copied a  
thousand files to the card & they checked out good.

Of the top of my head I don't know how big your "bs=1024 count=500960"  
file is - I would make a Bash script generate files c 5meg in size  
(maybe alternative between 3meg & 6meg?) and copy them to the card  
until it fills up. Then check them, delete them and do so again until  
all 1000 have been copied & checked.

Personally, I wouldn't be happy with one file being copied & stored  
ok, but I'd probably be happy if the card had proved itself to safely  
store 1000. And 3meg - 6meg is the size of the jpegs taken by my mum's  
camera. I would want to fill up the card, as flash cards try to avoid  
the bad blocks - if you don't fill it, you could be missing some duff  
block every time; once you've filled it you know that all the blocks  
are good. For some values of "good", that is - you don't know about  
longevity, and these cards are really just black boxes to us - we have  
no idea what magic goes on inside.

Personally, for my money, I don't know if I'd trust it. Depends what  
you're storing on it. MP3s for my phone? Sure - I have a backup at  
home. Moving files onto my PS3 or Wii, sure. For my camera? Maybe I'd  
be a bit cautious.

Stroller.

Re: [gentoo-user] testing a corrupt SD card

[Iain Buchanan]

On Fri, 2009-02-06 at 07:11 +0000, Stroller wrote:
> On 6 Feb 2009, at 05:28, Iain Buchanan wrote:
> > It's a Lexar Media 512Mb SD card, a couple of years old.  Yes I know I
> > can get a cheap 2Gb for <$20 but I'm more interested in the  
> > principle of
> > the test :)
> 
> I thought you could get then for < $5, but anyway....

probably in USD.  We (AUD) were approaching 1.00 before the exaggerated
crises, but now we're back to 0.645; and plus I needed one in a hurry,
so I couldn't order from a PC store which has reasonable prices and
instead had to go for a local and slightly more expensive retailer...

> > so I created a file:
> > dd if=/dev/urandom of=Desktop/random.img bs=1024 count=500960
> >
> > then copied it to the card, and then copied it back as  
> > random-2.img.  If
> > I md5sum the two files, they are identical:
> > $ md5sum random*
> > 9dcac25cfd8585be5939c0ff969de310  random-2.img
> > 9dcac25cfd8585be5939c0ff969de310  random.img
> >
> > Does that mean my memory card is good to go, or should I use some  
> > other
> > method of bad sector detection?
> 
> I'd be more or less happy with that methodology, had I copied a  
> thousand files to the card & they checked out good.
> 
> Of the top of my head I don't know how big your "bs=1024 count=500960" 

well, I got that from the free space on the card, using df and some
mathemagics, so it 100% fills the free space... however...

> file is - I would make a Bash script generate files c 5meg in size  
> (maybe alternative between 3meg & 6meg?) and copy them to the card  
> until it fills up. Then check them, delete them and do so again until  
> all 1000 have been copied & checked.

[snip]

however my method and your suggestion only fill up the free space, and
not the FAT for example, so there could be corruptions there, and given
I could see files but the names were nnnxxnnxnnn.ddxxc and so on, I
think it could have been a corrupt FAT?...

I should have made a file the size of the whole SD card, and just
written it to and read from the device a couple of times, overwriting
the partition table, and FAT.

> Personally, for my money, I don't know if I'd trust it. Depends what  
> you're storing on it. MP3s for my phone? Sure - I have a backup at  
> home. Moving files onto my PS3 or Wii, sure. For my camera? Maybe I'd  
> be a bit cautious.

Bought a new 2Gb.  Unfortunately I want a 512Mb card cause then I'm
forced to back it up often enough.

-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Men have a much better time of it than women; for one thing they marry later;
for another thing they die earlier.
		-- H.L. Mencken

[gentoo-user] Re: testing a corrupt SD card

[James]

Iain Buchanan <iaindb <at> netspace.net.au> writes:


> Does that mean my memory card is good to go, or should I use some other
> method of bad sector detection?

Hello Iain,


Hard to tell. Even repeated memory tests may/maynot find your gremlin,
i.e. bad memory. Also, just because it failed once, does not
mean the mem/card is bad. Transients occur giving false bit readings.
Sometimes you can delve into how the memory is actually made, but,
most vendors protect that info, with NDA and such, mostly smoke screens.

Here are a couple of links for your perusal:

http://en.wikipedia.org/wiki/Secure_Digital_card#SDHC

http://www.sandisk.com/Assets/File/OEM/Manuals/SD_SDIO_specsv1.pdf


Sorry, there is not a clear answer. Keep it for non critical needs,
upgrade to SDHC(fat 32) if your equipment supports that format.
Fat 32 on top of the memory, helps with (bit)error masking with
some enhance (undocumented) feature not part of fat 32. This is
what makes reverse engineering, complicated on SD memory. 
You may need to upgrade the firmware of your equipment to  support
newer SD standards (SD 1.1 and SD 2.0).


Good luck and good hunting (mate).....


James

Re: [gentoo-user] Re: testing a corrupt SD card

[Iain Buchanan]

On Fri, 2009-02-06 at 16:47 +0000, James wrote:
> Iain Buchanan <iaindb <at> netspace.net.au> writes:
> 
> 
> > Does that mean my memory card is good to go, or should I use some other
> > method of bad sector detection?
> 
> Hello Iain,

Hi James!

[snip]

> Here are a couple of links for your perusal:

Unfortunately I'm travelling, and the company here has a draconian
internet policy that doesn't allow much of anything.  I'll have to check
them out in a couple of weeks.

> Sorry, there is not a clear answer. Keep it for non critical needs,
> upgrade to SDHC(fat 32) if your equipment supports that format.

unfortunately not!

> Fat 32 on top of the memory, helps with (bit)error masking with
> some enhance (undocumented) feature not part of fat 32. This is
> what makes reverse engineering, complicated on SD memory. 
> You may need to upgrade the firmware of your equipment to  support
> newer SD standards (SD 1.1 and SD 2.0).

not much chance of that either (camera).  The more I look into it, the
solid state features (moving bad blocks around in firmware and hiding it
from the system) make me think it's time to throw it out anyway...

> Good luck and good hunting (mate).....

cheers, mate!
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

I might have gone to West Point, but I was too proud to speak to a congressman.
		-- Will Rogers

Re: [gentoo-user] testing a corrupt SD card

[Stroller]

On 6 Feb 2009, at 05:28, Iain Buchanan wrote:
> ...
> so I created a file:
> dd if=/dev/urandom of=Desktop/random.img bs=1024 count=500960

It has just occurred to me:

In the UK you can be imprisoned for failing to provide an encryption  
key corresponding to this file.

Stroller.

Re: [gentoo-user] testing a corrupt SD card

[Iain Buchanan]

On Fri, 2009-02-06 at 22:21 +0000, Stroller wrote:
> On 6 Feb 2009, at 05:28, Iain Buchanan wrote:
> > ...
> > so I created a file:
> > dd if=/dev/urandom of=Desktop/random.img bs=1024 count=500960
> 
> It has just occurred to me:
> 
> In the UK you can be imprisoned for failing to provide an encryption  
> key corresponding to this file.

are you joking? what's the story there?
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

< DanielS> still, throne of blood sounds like a movie about overfiend 
           and virgins or some crap
		-- in #debian-devel

Re: [gentoo-user] testing a corrupt SD card

[Stroller]

On 11 Feb 2009, at 00:54, Iain Buchanan wrote:
> On Fri, 2009-02-06 at 22:21 +0000, Stroller wrote:
>> On 6 Feb 2009, at 05:28, Iain Buchanan wrote:
>>> ...
>>> so I created a file:
>>> dd if=/dev/urandom of=Desktop/random.img bs=1024 count=500960
>>
>> It has just occurred to me:
>>
>> In the UK you can be imprisoned for failing to provide an encryption
>> key corresponding to this file.
>
> are you joking? what's the story there?

It is a facet of the Regulation of Investigatory Powers Act (RIPA),  
which was passed in 2000 but which only came into effect just over  
year ago.

     ... those served with a "Section 49" notice have to either
     make decryption keys available or put the data in an
     intelligible form for authorities. Failure to comply could
     mean a prison sentence of up to two years for cases not
     involving national security or five years for those that
     do.
     <http://www.infoworld.com/article/07/10/01/UK-encryption-disclosure-law-takes-effect_1.html 
 >


Under Part III of the act:

    If any person with the appropriate permission under Schedule
    2 believes, on reasonable grounds ... that a key to the
    protected information is in the possession of any person,
    ... the person with that permission may, by notice to the
    person whom he believes to have possession of the key,
    impose a disclosure requirement in respect of the protected
    information.
    <http://www.opsi.gov.uk/acts/acts2000/ukpga_20000023_en_8#pt3-pb1>

Because the generated file is indistinguishable from an encrypted file  
it may be reasonably be believed to be one. Especially if you are  
charged with a crime &/or use encryption for other purposes.

    In September 2003, Home Secretary David Blunkett announced
    wide-ranging extensions to the list of those entitled to see
    information collected under the RIPA. The list now includes
    jobcentres, local councils, and the Chief Inspector of
    Schools. Civil rights and privacy campaigners have dubbed
    these extensions a "snoopers' charter". At the passing of
    the act only nine organisations (including the police and
    security services) were allowed to invoke it, but as of
    2008, it was 792 organizations (including 474 councils).

    In April 2008, it became known that council officials in
    Dorset put three children and their parents under
    surveillance, governed by RIPA, at home and in their daily
    movements to check whether they lived in a particular school
    catchment area. This was in the context of rules which allow
    people who live in the school catchment area to enjoy
    advantages in obtaining a place at a popular school. The
    same council put fishermen under covert surveillance to
    check for the illegal harvesting of cockles and clams in
    ways that are regulated by RIPA. Other councils in the UK
    have conducted undercover operations regulated by RIPA
    against dog fouling and fly-tipping.
    <http://en.wikipedia.org/wiki/ 
Regulation_of_Investigatory_Powers_Act>

(The cases cited in the last paragraph surely apply to the RIPA's  
regulation of CCTV surveillance, rather than encryption, however I  
thought it relevant to illustrate how wide-ranging the use of this  
"anti-terrorism" act has become).

Stroller.

Re: [gentoo-user] testing a corrupt SD card

[Paul Hartman]

On Thu, Feb 5, 2009 at 11:28 PM, Iain Buchanan <iaindb@netspace.net.au> wrote:
> Hi all,
>
> recently my SD card just went bonkers.  Unfortunately I lost a lot of
> photos on it (backups are useless until the data actually gets to the
> backup...) but fortunately I was able to use a program to recover about
> 170 photos.
>
> Anyway, I don't know if it was just static, shock, dead card, or phase
> of the moon, so I would like to see if the card is good before I
> continue to use it.

With any kind of memory or storage device, I would stop using after
the first sign of a problem. My personal experience says it only gets
worse. :)

Lexar has a free program for recovering corrupted/deleted files from
their cards, did you use that? Or something linux-based like photorec?
Anyway, you wrote over it so it's too late now. :)

I would try any kind of "torture test" you can think of on it. Also be
sure to unmount/remove and reinsert it a few times. I have encountered
cards that only work 1 out of 4 times they are inserted into the card
reader, etc. Maybe the contacts are worn if it's a few years old?

Some (all?) memory cards do wear-leveling/load balancing so when you
write to sector 13 it might not be the same sector 13 as last time, so
doing any kind of repeated error testing might be difficult. In fact,
if the card detects bad spots it may simply hide them from you. There
may be programmatic ways to bypass the card's wear-leveling, but I
don't know how.

Good luck! In case you can't get it working reliably, Newegg has an
8gb CF card for $19.99 & free shipping :)

Paul

Re: [gentoo-user] testing a corrupt SD card

[Paul Hartman]

On Fri, Feb 6, 2009 at 7:36 PM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> Good luck! In case you can't get it working reliably, Newegg has an
> 8gb CF card for $19.99 & free shipping :)

Oops, never mind that part, I see you're in AU not US. My mistake! I
am an American after all, sometimes we forget other countries exist.
;)

Paul

Re: [gentoo-user] testing a corrupt SD card

[Iain Buchanan]

On Fri, 2009-02-06 at 19:36 -0600, Paul Hartman wrote:
> On Thu, Feb 5, 2009 at 11:28 PM, Iain Buchanan <iaindb@netspace.net.au> wrote:
> > Hi all,
> >
> > recently my SD card just went bonkers.  Unfortunately I lost a lot of
> > photos on it (backups are useless until the data actually gets to the
> > backup...) but fortunately I was able to use a program to recover about
> > 170 photos.
> >
> > Anyway, I don't know if it was just static, shock, dead card, or phase
> > of the moon, so I would like to see if the card is good before I
> > continue to use it.
> 
> With any kind of memory or storage device, I would stop using after
> the first sign of a problem. My personal experience says it only gets
> worse. :)
> 
> Lexar has a free program for recovering corrupted/deleted files from
> their cards, did you use that? Or something linux-based like photorec?
> Anyway, you wrote over it so it's too late now. :)

Now you tell me there are free versions?!  I ended up finding a photo
recovery tool which recovered the photos for me, but it wasn't "free".
Needless to say I didn't pay for it, and I deleted it straight away.

I'll check out photorec next time.  I'm having a hard time finding info
about it though (see previous email about draconian internet access).

Is there a general linux version of FAT recovery tools available
somewhere?  I couldn't find one.

[sinp]

> Some (all?) memory cards do wear-leveling/load balancing

This is what I was afraid of.

thanks,
-- 
Iain Buchanan <iaindb at netspace dot net dot au>

Beware the new TTY code!

Re: [gentoo-user] testing a corrupt SD card

[Paul Hartman]

On Tue, Feb 10, 2009 at 6:59 PM, Iain Buchanan <iaindb@netspace.net.au> wrote:
> On Fri, 2009-02-06 at 19:36 -0600, Paul Hartman wrote:
>> On Thu, Feb 5, 2009 at 11:28 PM, Iain Buchanan <iaindb@netspace.net.au> wrote:
>> > Hi all,
>> >
>> > recently my SD card just went bonkers.  Unfortunately I lost a lot of
>> > photos on it (backups are useless until the data actually gets to the
>> > backup...) but fortunately I was able to use a program to recover about
>> > 170 photos.
>> >
>> > Anyway, I don't know if it was just static, shock, dead card, or phase
>> > of the moon, so I would like to see if the card is good before I
>> > continue to use it.
>>
>> With any kind of memory or storage device, I would stop using after
>> the first sign of a problem. My personal experience says it only gets
>> worse. :)
>>
>> Lexar has a free program for recovering corrupted/deleted files from
>> their cards, did you use that? Or something linux-based like photorec?
>> Anyway, you wrote over it so it's too late now. :)
>
> Now you tell me there are free versions?!  I ended up finding a photo
> recovery tool which recovered the photos for me, but it wasn't "free".
> Needless to say I didn't pay for it, and I deleted it straight away.

Actually, the Lexar Image Rescue software costs USD$33.95, but "will
be available for free download with purchase of any new 2007 Lexar
Professional or Platinum II line CompactFlash or Secure Digital memory
card."

Also, I just learned that Lexar will do professional data recovery for
FREE on Lexar cards (if you mail them in). Apparently it is
unadvertised benefit geared mostly toward professional photographers
who cannot afford to lose their work, so if you had files other than
photos on the card they probably wouldn't deal with them. I don't know
if Lexar AU could do it or if you'd need to send it to the USA. Let's
hope you never have to deal with that situation again. :)

> I'll check out photorec next time.  I'm having a hard time finding info
> about it though (see previous email about draconian internet access).

A description from the web:

PhotoRec is file data recovery software designed to recover lost files
including video, documents and archives from Hard Disks and CDRom and
lost pictures (thus, its 'Photo Recovery' name) from digital camera
memory. PhotoRec ignores the filesystem and goes after the underlying
data, so it will still work even if your media's filesystem has been
severely damaged or re-formatted.

PhotoRec is free, this open source multi-platform application is
distributed under GNU Public License. PhotoRec is a companion program
to TestDisk, an app for recovering lost partitions on a wide variety
of filesystems and making non-bootable disks bootable again. You can
download them from this link.

For more safety, PhotoRec uses read-only access to handle the drive or
memory support you are about to recover lost data from. Important: As
soon as a pic or file is accidentally deleted, or you discover any
missing, do NOT save any more pics or files to that memory device or
hard disk drive; otherwise you may overwrite your lost data. This
means that even using PhotoRec, you must not choose to write the
recovered files to the same partition they were stored on.

> Is there a general linux version of FAT recovery tools available
> somewhere?  I couldn't find one.

Well, other than fsck I'm familiar with DFSee. It is not free, but it
is shareware with a trial period (good if you only need to use it once
in a pinch), and it has a Linux (and DOS,Windows, Mac and OS/2)
version as well as bootable ISO. I've used it for many years and had
some success stories. I registered it back in the olden days (version
2 or 3 or so), now it's up to v9. http://www.dfsee.com/

Again, hoping you never have to use such a thing!
Paul