[gentoo-user] Problems with Nvidia fake raid array

[gentoo-user] Problems with Nvidia fake raid array

[Jeff Cranmer]

Hi all,

After cleaning off my Opensuse O.S. and installing Gentoo, I'm having
trouble getting my 3-disk nvidia SATA raid5 array back on line.

The gentoo OS is on a separate non-raid IDE disk, and I can see the
three individual disks which make up the raid array (/dev/sda, /dev/sdb
and /dev/sdc).  Unfortunately, the system does not seem to be able to
detect the raid array, and dmesg shows no md disks detected or mounted.

There are a few guides on line for setting up a system which boots to a
raid array, but I haven't found any guides for simply mounting a raided
disk.  I think I've got all the kernel settings right, and the raid
array was working before I cleared out the IDE disk.  I know that the
Nvidia array isn't a true hardware raid array, but it's a data disk
only, and while I have a reasonably recent backup, I'm not keen on
re-formatting it and setting up a kernel-based raid array.

Any suggestions or pointers gratefully received.  Thanks in advance.

Jeff

Re: [gentoo-user] Problems with Nvidia fake raid array

[Michael Orlitzky]

On 07/18/2011 09:26 PM, Jeff Cranmer wrote:
> Hi all,
> 
> After cleaning off my Opensuse O.S. and installing Gentoo, I'm having
> trouble getting my 3-disk nvidia SATA raid5 array back on line.
> 
> The gentoo OS is on a separate non-raid IDE disk, and I can see the
> three individual disks which make up the raid array (/dev/sda, /dev/sdb
> and /dev/sdc).  Unfortunately, the system does not seem to be able to
> detect the raid array, and dmesg shows no md disks detected or mounted.

Make sure your kernel supports RAID, and RAID5 (they're separate
options). Then emerge mdadm. Once you get it up and running once, you
can dump the current config to /etc/mdadm.conf so you don't have to
assemble it again. Then add mdadm to the boot runlevel.

# mdadm --assemble --help
Usage: mdadm --assemble device options...
       mdadm --assemble --scan options...

This usage assembles one or more raid arrays from pre-existing
components. For each array, mdadm needs to know the md device, the
identity of the array, and a number of sub devices. These can be found
in a number of ways.

The md device is either given on the command line or is found listed
in the config file. The array identity is determined either from the
--uuid or --super-minor commandline arguments, from the config file,
or from the first component device on the command line.

The different combinations of these are as follows:
If the --scan option is not given, then only devices and identities
listed on the command line are considered.

The first device will be the array device, and the remainder will be
examined when looking for components.

If an explicit identity is given with --uuid or --super-minor, then
only devices with a superblock which matches that identity is
considered, otherwise every device listed is considered.

If the --scan option is given, and no devices are listed, then
every array listed in the config file is considered for assembly.
The identity of candidate devices are determined from the config file.

If the --scan option is given as well as one or more devices, then
Those devices are md devices that are to be assembled. Their identity
and components are determined from the config file.

If mdadm can not find all of the components for an array, it will
assemble it but not activate it unless --run or --scan is given. To
preserve this behaviour even with --scan, add --no-degraded. Note that
"all of the components" means as many as were present the last time the
array was running as recorded in the superblock. If the array was
already degraded, and the missing device is not a new problem, it will
still be assembled. It is only newly missing devices that cause the
array not to be started.

Re: [gentoo-user] Problems with Nvidia fake raid array

[Jeff Cranmer]

On Mon, 2011-07-18 at 22:29 -0400, Michael Orlitzky wrote:
> 
> Make sure your kernel supports RAID, and RAID5 (they're separate
> options). Then emerge mdadm. Once you get it up and running once, you
> can dump the current config to /etc/mdadm.conf so you don't have to
> assemble it again. Then add mdadm to the boot runlevel.
> 
I'm Ok so far - Raid and Raid5 options are both already compiled into
the kernel, and mdadm is in the boot runlevel.

> # mdadm --assemble --help
> Usage: mdadm --assemble device options...
>        mdadm --assemble --scan options...
> 
> This usage assembles one or more raid arrays from pre-existing
> components. For each array, mdadm needs to know the md device, the
> identity of the array, and a number of sub devices. These can be found
> in a number of ways.

> The md device is either given on the command line or is found listed
> in the config file. The array identity is determined either from the
> --uuid or --super-minor commandline arguments, from the config file,
> or from the first component device on the command line.
> 
> The different combinations of these are as follows:
> If the --scan option is not given, then only devices and identities
> listed on the command line are considered.
> 
> The first device will be the array device, and the remainder will be
> examined when looking for components.
> 
> If an explicit identity is given with --uuid or --super-minor, then
> only devices with a superblock which matches that identity is
> considered, otherwise every device listed is considered.
> 
> If the --scan option is given, and no devices are listed, then
> every array listed in the config file is considered for assembly.
> The identity of candidate devices are determined from the config file.
> 
> If the --scan option is given as well as one or more devices, then
> Those devices are md devices that are to be assembled. Their identity
> and components are determined from the config file.
> 
> If mdadm can not find all of the components for an array, it will
> assemble it but not activate it unless --run or --scan is given. To
> preserve this behaviour even with --scan, add --no-degraded. Note that
> "all of the components" means as many as were present the last time the
> array was running as recorded in the superblock. If the array was
> already degraded, and the missing device is not a new problem, it will
> still be assembled. It is only newly missing devices that cause the
> array not to be started.

Pardon my additional questions before taking the plunge here.  

So, given that I have three devices, /dev/sda, /dev/sdb and /dev/sdc, if
I run the command mdadm --assemble --scan, would this find all the
components and create a /dev/md0 disk without damaging the contents of
the original RAID array?

The only item in /dev/mapper is th default 'control' entry.  There is
a /dev/md0 item already listed, but presently when I try to mount it, it
reports that it is unable to read the superblock.  Would the command
above fix this?

Where is the config file mentioned in your e-mail, and do I need to edit
it first to add the three raid disks?

Thanks

Jeff

Re: [gentoo-user] Problems with Nvidia fake raid array

[Michael Orlitzky]

On 07/18/2011 11:08 PM, Jeff Cranmer wrote:
> 
> 
> Pardon my additional questions before taking the plunge here.  
> 
> So, given that I have three devices, /dev/sda, /dev/sdb and /dev/sdc, if
> I run the command mdadm --assemble --scan, would this find all the
> components and create a /dev/md0 disk without damaging the contents of
> the original RAID array?

If you've got the space and time, a backup can't hurt. Using --scan will
make it check the config file, but right now, there's probably nothing
useful in it. This looks like what you want to do to me:

  If the --scan option is not given, then only devices and identities
  listed on the command line are considered.

  The first device will be the array device, and the remainder will be
  examined when looking for components.

but I'd figure out where that md0 is coming from (below) first.


> The only item in /dev/mapper is th default 'control' entry.  There is
> a /dev/md0 item already listed, but presently when I try to mount it, it
> reports that it is unable to read the superblock.  Would the command
> above fix this?

Depends. Where'd the md0 come from? You probably have something in your
logs or dmesg, unless that device was created manually on your old system.


> Where is the config file mentioned in your e-mail, and do I need to edit
> it first to add the three raid disks?

It's /etc/mdadm.conf. You don't need it to create or use the array, but
you'll want to run mdadm when the machine boots and the config file
tells it what to do. Once the array is working, you can just do,

  mdadm --detail --scan >> /etc/mdadm.conf

to populate it. I guess also check to make sure there's no default crap
in there these days.

Re: [gentoo-user] Problems with Nvidia fake raid array

[Volker Armin Hemmann]

On Tuesday 19 July 2011 16:36:20 you wrote:
> On Tuesday 19 July 2011 09:06:05 Michael Orlitzky wrote:
> > On 07/18/2011 11:08 PM, Jeff Cranmer wrote:
> > > Pardon my additional questions before taking the plunge here.
> > > 
> > > So, given that I have three devices, /dev/sda, /dev/sdb and
> > > /dev/sdc, if I run the command mdadm --assemble --scan, would this
> > > find all the components and create a /dev/md0 disk without damaging
> > > the contents of the original RAID array?
> > 
> > If you've got the space and time, a backup can't hurt. Using --scan will
> > make it check the config file, but right now, there's probably nothing
> > 
> > useful in it. This looks like what you want to do to me:
> >   If the --scan option is not given, then only devices and identities
> >   listed on the command line are considered.
> >   
> >   The first device will be the array device, and the remainder will be
> >   examined when looking for components.
> > 
> > but I'd figure out where that md0 is coming from (below) first.
> > 
> > > The only item in /dev/mapper is th default 'control' entry.  There
> > > is
> > > a /dev/md0 item already listed, but presently when I try to mount
> > > it, it reports that it is unable to read the superblock.  Would the
> > > command above fix this?
> > 
> > Depends. Where'd the md0 come from? You probably have something in your
> > logs or dmesg, unless that device was created manually on your old
> > system.> 
> > > Where is the config file mentioned in your e-mail, and do I need to
> > > edit it first to add the three raid disks?
> > 
> > It's /etc/mdadm.conf. You don't need it to create or use the array, but
> > you'll want to run mdadm when the machine boots and the config file
> > tells it what to do. Once the array is working, you can just do,
> > 
> >   mdadm --detail --scan >> /etc/mdadm.conf
> > 
> > to populate it. I guess also check to make sure there's no default crap
> > in there these days.
> 
> mdadm? that is for linux software raid.
> 
> He wants to use nvidia fakeraid. He needs dmraid for that.
> 
> Build in the dmraid support.
> emerge device-mapper
> emerge dmraid
> add device-mapper to boot runlevel
> done
> 
> If I remember correctly
> 
> oh yeah - nvidia fakeraid is pretty fragile. You should plan to move away
> from it.

addendum: just saw that mdadm >=3 should be able to handle fake raid.

My bad.

-- 
#163933

Re: [gentoo-user] Problems with Nvidia fake raid array

[Jeff Cranmer]

On Tue, 2011-07-19 at 09:06 -0400, Michael Orlitzky wrote:
> On 07/18/2011 11:08 PM, Jeff Cranmer wrote:
> > 
> > 
> > Pardon my additional questions before taking the plunge here.  
> > 
> > So, given that I have three devices, /dev/sda, /dev/sdb and /dev/sdc, if
> > I run the command mdadm --assemble --scan, would this find all the
> > components and create a /dev/md0 disk without damaging the contents of
> > the original RAID array?
> 
> If you've got the space and time, a backup can't hurt. Using --scan will
> make it check the config file, but right now, there's probably nothing
> useful in it. This looks like what you want to do to me:
> 
>   If the --scan option is not given, then only devices and identities
>   listed on the command line are considered.
> 
>   The first device will be the array device, and the remainder will be
>   examined when looking for components.
> 
> but I'd figure out where that md0 is coming from (below) first.
> 
When I tried mdadm --assemble --scan with nothing uncommented in the
configuration file, I got
mdadm: No arrays found in config file or automatically.
Typing dmesg | grep md0 returned no lines.

There are a couple of lines in dmesg when I run dmesg | grep md:, but
they read
md: linear personality registered for level -1
md: raid0 personality registered for level 0
md: raid1 personality registered for level 1
md: raid10 personality registered for level 10
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md: Waiting for all devices to be available before autodetect
md: If you don't use raid, use raid=noautodetect
md: Autodetecting RAID arrays
md: Scanned 0 and added 0 devices
md: autorun...
md: ... autorun DONE.

I think this means that raid5 is set up correctly in the kernel, but it
can't find the raid array.

Next I tried adding a line to the config file:

DEVICE /dev/sda /dev/sdb /dev/sdc
mdadm --assemble --scan returned the same results as before

Next, I tried commenting out the previously added DEVICE line, and
adding
ARRAY /dev/md0 devices=/dev/sda,/dev/sdb,/dev/sdc

mdadm --assemble --scan returns something different
mdadm: /dev/sdb has no superblock - assembly aborted.
> 
> > The only item in /dev/mapper is th default 'control' entry.  There is
> > a /dev/md0 item already listed, but presently when I try to mount it, it
> > reports that it is unable to read the superblock.  Would the command
> > above fix this?
> 
> Depends. Where'd the md0 come from? You probably have something in your
> logs or dmesg, unless that device was created manually on your old system.
> 
> 
> > Where is the config file mentioned in your e-mail, and do I need to edit
> > it first to add the three raid disks?
> 
> It's /etc/mdadm.conf. You don't need it to create or use the array, but
> you'll want to run mdadm when the machine boots and the config file
> tells it what to do. Once the array is working, you can just do,
> 
>   mdadm --detail --scan >> /etc/mdadm.conf
> 
mdadm --detail --scan returns no output.

Also, I just checked /dev and md0 is now gone from the list.

Since there are also /dev/sg0, /dev/sg1 and /dev/sg1, I also tried those
instead of /dev/sda, /dev/sdb and /dev/sdc in the ARRAY line, but mdadm
--assemble --scan returned no output

I tried re-booting, but /dev/md0 is now permanently gone.

Does this give you any ideas what I can try next??

Thanks

Jeff

Re: [gentoo-user] Problems with Nvidia fake raid array

[Jeff Cranmer]

Is there anyone who can help me recover my raid array?

On Wed, 2011-07-20 at 20:43 -0400, Jeff Cranmer wrote:
> On Tue, 2011-07-19 at 09:06 -0400, Michael Orlitzky wrote:
> > On 07/18/2011 11:08 PM, Jeff Cranmer wrote:
> > > 
> > > 
> > > Pardon my additional questions before taking the plunge here.  
> > > 
> > > So, given that I have three devices, /dev/sda, /dev/sdb and /dev/sdc, if
> > > I run the command mdadm --assemble --scan, would this find all the
> > > components and create a /dev/md0 disk without damaging the contents of
> > > the original RAID array?
> > 
> > If you've got the space and time, a backup can't hurt. Using --scan will
> > make it check the config file, but right now, there's probably nothing
> > useful in it. This looks like what you want to do to me:
> > 
> >   If the --scan option is not given, then only devices and identities
> >   listed on the command line are considered.
> > 
> >   The first device will be the array device, and the remainder will be
> >   examined when looking for components.
> > 
> > but I'd figure out where that md0 is coming from (below) first.
> > 
> When I tried mdadm --assemble --scan with nothing uncommented in the
> configuration file, I got
> mdadm: No arrays found in config file or automatically.
> Typing dmesg | grep md0 returned no lines.
> 
> There are a couple of lines in dmesg when I run dmesg | grep md:, but
> they read
> md: linear personality registered for level -1
> md: raid0 personality registered for level 0
> md: raid1 personality registered for level 1
> md: raid10 personality registered for level 10
> md: raid6 personality registered for level 6
> md: raid5 personality registered for level 5
> md: raid4 personality registered for level 4
> md: Waiting for all devices to be available before autodetect
> md: If you don't use raid, use raid=noautodetect
> md: Autodetecting RAID arrays
> md: Scanned 0 and added 0 devices
> md: autorun...
> md: ... autorun DONE.
> 
> I think this means that raid5 is set up correctly in the kernel, but it
> can't find the raid array.
> 
> Next I tried adding a line to the config file:
> 
> DEVICE /dev/sda /dev/sdb /dev/sdc
> mdadm --assemble --scan returned the same results as before
> 
> Next, I tried commenting out the previously added DEVICE line, and
> adding
> ARRAY /dev/md0 devices=/dev/sda,/dev/sdb,/dev/sdc
> 
> mdadm --assemble --scan returns something different
> mdadm: /dev/sdb has no superblock - assembly aborted.
> > 
> > > The only item in /dev/mapper is th default 'control' entry.  There is
> > > a /dev/md0 item already listed, but presently when I try to mount it, it
> > > reports that it is unable to read the superblock.  Would the command
> > > above fix this?
> > 
> > Depends. Where'd the md0 come from? You probably have something in your
> > logs or dmesg, unless that device was created manually on your old system.
> > 
> > 
> > > Where is the config file mentioned in your e-mail, and do I need to edit
> > > it first to add the three raid disks?
> > 
> > It's /etc/mdadm.conf. You don't need it to create or use the array, but
> > you'll want to run mdadm when the machine boots and the config file
> > tells it what to do. Once the array is working, you can just do,
> > 
> >   mdadm --detail --scan >> /etc/mdadm.conf
> > 
> mdadm --detail --scan returns no output.
> 
> Also, I just checked /dev and md0 is now gone from the list.
> 
> Since there are also /dev/sg0, /dev/sg1 and /dev/sg1, I also tried those
> instead of /dev/sda, /dev/sdb and /dev/sdc in the ARRAY line, but mdadm
> --assemble --scan returned no output
> 
> I tried re-booting, but /dev/md0 is now permanently gone.
> 
> Does this give you any ideas what I can try next??
> 
> Thanks
> 
> Jeff
> 
> 
> 

Re: [gentoo-user] Problems with Nvidia fake raid array

[Michael Orlitzky]

On 07/22/11 21:56, Jeff Cranmer wrote:
> Is there anyone who can help me recover my raid array?
> 
>>
>> Next, I tried commenting out the previously added DEVICE line, and
>> adding
>> ARRAY /dev/md0 devices=/dev/sda,/dev/sdb,/dev/sdc
>>
>> mdadm --assemble --scan returns something different
>> mdadm: /dev/sdb has no superblock - assembly aborted.

I think this one should have worked? It seems to have found the
superblock on /dev/sda, at least.

Anyway, I imagine everyone (myself included) is afraid to tell you to do
anything at this point that might trash your data. My advice now would
be to put it back where it worked, and make a backup.

Re: [gentoo-user] Problems with Nvidia fake raid array

[Jeff Cranmer]

On Mon, 2011-07-25 at 10:45 -0400, Michael Orlitzky wrote:
> On 07/22/11 21:56, Jeff Cranmer wrote:
> > Is there anyone who can help me recover my raid array?
> > 
> >>
> >> Next, I tried commenting out the previously added DEVICE line, and
> >> adding
> >> ARRAY /dev/md0 devices=/dev/sda,/dev/sdb,/dev/sdc
> >>
> >> mdadm --assemble --scan returns something different
> >> mdadm: /dev/sdb has no superblock - assembly aborted.
> 
> I think this one should have worked? It seems to have found the
> superblock on /dev/sda, at least.
> 
> Anyway, I imagine everyone (myself included) is afraid to tell you to do
> anything at this point that might trash your data. My advice now would
> be to put it back where it worked, and make a backup.
> 
I already

Re: [gentoo-user] Problems with Nvidia fake raid array

[Jeff Cranmer]

On Mon, 2011-07-25 at 10:45 -0400, Michael Orlitzky wrote:
> On 07/22/11 21:56, Jeff Cranmer wrote:
> > Is there anyone who can help me recover my raid array?
> > 
> >>
> >> Next, I tried commenting out the previously added DEVICE line, and
> >> adding
> >> ARRAY /dev/md0 devices=/dev/sda,/dev/sdb,/dev/sdc
> >>
> >> mdadm --assemble --scan returns something different
> >> mdadm: /dev/sdb has no superblock - assembly aborted.
> 
> I think this one should have worked? It seems to have found the
> superblock on /dev/sda, at least.
> 
> Anyway, I imagine everyone (myself included) is afraid to tell you to do
> anything at this point that might trash your data. My advice now would
> be to put it back where it worked, and make a backup.
> 
I already have a backup of the data (much of it in multiple locations).
If it gets trashed, I can recover all critical data.  Right now, I just
want to get the storage space back.

Jeff

Re: [gentoo-user] Problems with Nvidia fake raid array

[Michael Orlitzky]

On 07/25/2011 08:00 PM, Jeff Cranmer wrote:
> On Mon, 2011-07-25 at 10:45 -0400, Michael Orlitzky wrote:
>> On 07/22/11 21:56, Jeff Cranmer wrote:
>>> Is there anyone who can help me recover my raid array?
>>>
>>>>
>>>> Next, I tried commenting out the previously added DEVICE line, and
>>>> adding
>>>> ARRAY /dev/md0 devices=/dev/sda,/dev/sdb,/dev/sdc
>>>>
>>>> mdadm --assemble --scan returns something different
>>>> mdadm: /dev/sdb has no superblock - assembly aborted.
>>
>> I think this one should have worked? It seems to have found the
>> superblock on /dev/sda, at least.
>>
>> Anyway, I imagine everyone (myself included) is afraid to tell you to do
>> anything at this point that might trash your data. My advice now would
>> be to put it back where it worked, and make a backup.
>>
> I already have a backup of the data (much of it in multiple locations).
> If it gets trashed, I can recover all critical data.  Right now, I just
> want to get the storage space back.

If you have the data elsewhere, you can just wipe the disks and recreate
the array, right?

I usually use this rather than read the mdadm --create manpage:

http://en.gentoo-wiki.com/wiki/Software_RAID_Install#Setting_up_the_RAID

If you want it to work in Windows... Well, I'm not sure how to *create*
a fakeraid with mdadm. The last time I tried (6 years ago?), dmraid was
the only way. In that case I would spend $25 on a bargain bin Adaptec
card and consider it an investment in future-you's mental health.

Re: Re: [gentoo-user] Problems with Nvidia fake raid array

[Daniel Frey]

On 01/-10/37 11:59, Michael Orlitzky wrote:
> 
> I think this one should have worked? It seems to have found the
> superblock on /dev/sda, at least.
> 
> Anyway, I imagine everyone (myself included) is afraid to tell you to do
> anything at this point that might trash your data. My advice now would
> be to put it back where it worked, and make a backup.

I'm just going through this myself. As far as I know mdadm does *not*
support nvraid. It does support imsm, or intel raid, which I'm in the
process of setting up on my workstation.

I can't find anything in the docs regarding mdadm working with nvraid,
you should be trying dmraid for that.

If all you have is /dev/control and you are not using a dmraid supported
kernel (genkernel requires dodmraid to find and assemble arrays) then
execute `dmraid -ay` and check dmesg and /dev/mapper for contents.

Dan

Re: Re: [gentoo-user] Problems with Nvidia fake raid array

[Jeff Cranmer]

On Tue, 2011-07-26 at 16:55 -0700, Daniel Frey wrote:
> On 01/-10/37 11:59, Michael Orlitzky wrote:
> > 
> > I think this one should have worked? It seems to have found the
> > superblock on /dev/sda, at least.
> > 
> > Anyway, I imagine everyone (myself included) is afraid to tell you to do
> > anything at this point that might trash your data. My advice now would
> > be to put it back where it worked, and make a backup.
> 
> I'm just going through this myself. As far as I know mdadm does *not*
> support nvraid. It does support imsm, or intel raid, which I'm in the
> process of setting up on my workstation.
> 
> I can't find anything in the docs regarding mdadm working with nvraid,
> you should be trying dmraid for that.
> 
> If all you have is /dev/control and you are not using a dmraid supported
> kernel (genkernel requires dodmraid to find and assemble arrays) then
> execute `dmraid -ay` and check dmesg and /dev/mapper for contents.
> 
> Dan
> 
Thanks Dan,

I'm not using genkernel at the moment.  I'll try those steps and let you
know how it goes.  The Raid array worked on the OpenSuse operating
system that I blew away to install gentoo, so I should be able to
resurrect it without wiping everything out.

Jeff