[gentoo-user] mdadm fails to add drive to arry

[gentoo-user] mdadm fails to add drive to arry

[Richard Broersma Jr]

I recently had a drive failure in with a Raid 1 setup.  However, for testing purposes I tried to
re-add the drive to the array after I removed it.  However I ran into the following error message.

~ # mdadm /dev/md4 --add /dev/hdj1
mdadm: add new device failed for /dev/hdj1 as 2: Invalid argument

Where is the invalid argument?  Or is this error message you get when it is time to buy a new hard
drive :).

I verified my syntax from that man page and :
http://www.linuxdevcenter.com/pub/a/linux/2002/12/05/RAID.html?page=2
under the section: Managing Arrays

Regards,

Richard Broesma Jr.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] mdadm fails to add drive to arry

[Richard Fish]

On 9/1/06, Richard Broersma Jr <rabroersma@yahoo.com> wrote:
> I recently had a drive failure in with a Raid 1 setup.  However, for testing purposes I tried to
> re-add the drive to the array after I removed it.  However I ran into the following error message.
>
> ~ # mdadm /dev/md4 --add /dev/hdj1
> mdadm: add new device failed for /dev/hdj1 as 2: Invalid argument
>
> Where is the invalid argument?  Or is this error message you get when it is time to buy a new hard

I think you forgot to specify the mode.  From man mdadm:

"""
SYNOPSIS
       mdadm [mode] <raiddevice> [options] <component-devices>
"""

and

"""
      If  a  device  is  given  before any options, or if the first option is
       --add, --fail, or --remove, then the MANAGE mode is  assume.   Anything
       other than these will CAUSE THE MISC MODE TO BE ASSUMED.
"""
(emphasis added)

So I think you need "mdadm manage /dev/md4 --add /dev/hdj1".  Or
"mdadm --add /dev/md4 /dev/hdj1".

This isn't based on an actual test, just reading the documentation, so I HTH.

-Richard
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] mdadm fails to add drive to arry

[Richard Fish]

On 9/1/06, Richard Fish <bigfish@asmallpond.org> wrote:
> > Where is the invalid argument?  Or is this error message you get when it is time to buy a new hard
>
> I think you forgot to specify the mode.  From man mdadm:


I should really learn to read better....sorry.

Did you remove the faulty device first?  IIRC, raid devices start
numbering at 0, so it looks like this is trying to add a third device
(#2), instead of replacing #0 or #1.

-Richard
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] mdadm fails to add drive to arry

[Richard Broersma Jr]

> I think you forgot to specify the mode.  From man mdadm:
> SYNOPSIS
>        mdadm [mode] <raiddevice> [options] <component-devices>
>       If  a  device  is  given  before any options, or if the first option is
>        --add, --fail, or --remove, then the MANAGE mode is  assume.   Anything
>        other than these will CAUSE THE MISC MODE TO BE ASSUMED.
> (emphasis added)
> So I think you need "mdadm manage /dev/md4 --add /dev/hdj1".  Or
> "mdadm --add /dev/md4 /dev/hdj1".

I appreciate the feed back.  Here is what I am getting.

~ # mdadm manage /dev/md4 --add /dev/hdj1
mdadm: An option must be given to set the mode before a second device is listed

~ # mdadm --add /dev/md4 /dev/hdj1
mdadm: add new device failed for /dev/hdj1 as 2: Invalid argument

and for kicks :)
~ # mdadm manage --add /dev/md4 /dev/hdj1
mdadm: error opening manage: No such file or directory
~ # mdadm --add manage /dev/md4 /dev/hdj1
mdadm: error opening manage: No such file or directory

Regards,

Richard Broersma Jr.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] mdadm fails to add drive to arry

[Richard Broersma Jr]

> I should really learn to read better....sorry.

No problem :-)

> Did you remove the faulty device first?
yup.
Personalities : [raid1] [raid10] [multipath]
md4 : active raid1 hdg1[1]
      293049600 blocks [2/1] [_U]


  IIRC, raid devices start
> numbering at 0, so it looks like this is trying to add a third device
> (#2), instead of replacing #0 or #1.

I am not exact sure on this point.  Are you refering to the meta device?

Regards,

Richard Broersma Jr.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] mdadm fails to add drive to arry

[Richard Fish]

On 9/1/06, Richard Broersma Jr <rabroersma@yahoo.com> wrote:
> > Did you remove the faulty device first?
> yup.
> Personalities : [raid1] [raid10] [multipath]
> md4 : active raid1 hdg1[1]
>       293049600 blocks [2/1] [_U]
>
>
>   IIRC, raid devices start
> > numbering at 0, so it looks like this is trying to add a third device
> > (#2), instead of replacing #0 or #1.
>
> I am not exact sure on this point.  Are you refering to the meta device?

No, I mean the device numbers.  I'm now at home, so I can test a bit
with my AMD64 box.  When I create a raid1 array there, /proc/mdstat
contains:

Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid1 sdb2[1] sda2[0]
      9775488 blocks [2/2] [UU]
      [>....................]  resync =  2.7% (271552/9775488)
finish=2.3min speed=67888K/sec

Notice the 2 device nodes....[1] and [0].  I think your setup is
trying to add a [2], which probably doesn't work because the array was
created with --raid-devices=2.  This is only a guess though, based on
the " as 2" part of the error message.

Ok, so lets say I create an array with a missing element:

~ > mdadm --create --level=1 --raid-devices=2 /dev/md0 missing /dev/sdb2
mdadm: /dev/sdb2 appears to be part of a raid array:
    level=raid1 devices=2 ctime=Sat Sep  2 02:07:13 2006
Continue creating array? yes
mdadm: array /dev/md0 started.
~ > cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid1 sdb2[1]
      9775488 blocks [2/1] [_U]

unused devices: <none>

Ok, so now let me try repairing it...

~ > mdadm --add /dev/md0 /dev/sda2
mdadm: added /dev/sda2
~ > cat /proc/mdstat
Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid1 sda2[2] sdb2[1]
      9775488 blocks [2/1] [_U]
      [>....................]  recovery =  3.4% (340928/9775488)
finish=2.3min speed=68185K/sec

unused devices: <none>

Hmm, works fine.... :-(

Ok, maybe stupid questions time.

/dev/hdj1 does exist, right?
Is this the same drive that was once part of the array?  Or a new drive?
If new, is the partition at least as large as /dev/hdg1?  (double
check with fdisk output).
If the same drive, does it work if you do "mdadm --zero-superblock
/dev/hdj1" first?
What does "mdadm --examine /dev/hdj1" report?  How about for /dev/hdg1?

I know that is a lot of questions to ask, but I don't see anything
obviously wrong at this point...so I am grasping at straws.

-Richard
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] mdadm fails to add drive to arry

[Richard Broersma Jr]

> Ok, maybe stupid questions time.
> 
> /dev/hdj1 does exist, right?

yes

> Is this the same drive that was once part of the array? 

It is existing.  But possible it may have legitamatly failed. Here is a log generated today when I
tried to add it back into the array.
Sep  2 09:11:02 [kernel] end_request: I/O error, dev hdj, sector 586099263
Sep  2 09:11:02 [kernel] Buffer I/O error on device hdj1, logical block 146524800
...
Sep  2 09:11:02 [kernel] lost page write due to I/O error on hdj1
Sep  2 09:11:02 [kernel] end_request: I/O error, dev hdj, sector 586099263
Sep  2 09:11:02 [kernel] md: disabled device hdj1, could not read superblock.
Sep  2 09:11:02 [kernel] md: hdj1 has invalid sb, not importing!
Sep  2 09:11:02 [kernel] md: md_import_device returned -22


> If the same drive, does it work if you do "mdadm --zero-superblock
> /dev/hdj1" first?

~ # mdadm --zero-superblock /dev/hdj1
mdadm: Unrecognised md component device - /dev/hdj1



> What does "mdadm --examine /dev/hdj1" report?  How about for /dev/hdg1?


~ # mdadm --zero-superblock /dev/hdj1
mdadm: Unrecognised md component device - /dev/hdj1

~ # mdadm --examine /dev/hdj1
mdadm: No md superblock detected on /dev/hdj1.
db_server01 ~ # mdadm --examine /dev/hdg1
/dev/hdg1:
          Magic : a92b4efc
        Version : 00.90.03
           UUID : d77079f9:c084dedf:ba141e09:96726a79
  Creation Time : Tue May 30 19:48:40 2006
     Raid Level : raid1
    Device Size : 293049600 (279.47 GiB 300.08 GB)
     Array Size : 293049600 (279.47 GiB 300.08 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 4

    Update Time : Fri Sep  1 20:52:33 2006
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 2cbf93d9 - correct
         Events : 0.1402072


      Number   Major   Minor   RaidDevice State
this     1      34        1        1      active sync   /dev/hdg1

   0     0       0        0        0      removed
   1     1      34        1        1      active sync   /dev/hdg1

Does all of this suggest a complete and total drive failure? :-)  I orginally built this md4 array
as a Raid 5 from three of the cheepest drives that I could find (Maxtors).  Then a couple of
months later, one drive died.  So I converted the remaining to into a raid 1.  Now this one seems
to have died.  I guess I am learning my leason about buying cheep hardware.  Fortunately I have a
back-up of everything on this array.
 
 
Regards,

Richard Broersma Jr.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] mdadm fails to add drive to arry

[Richard Fish]

On 9/2/06, Richard Broersma Jr <rabroersma@yahoo.com> wrote:
> It is existing.  But possible it may have legitamatly failed. Here is a log generated today when I
> tried to add it back into the array.
> Sep  2 09:11:02 [kernel] end_request: I/O error, dev hdj, sector 586099263
> Sep  2 09:11:02 [kernel] Buffer I/O error on device hdj1, logical block 146524800
> ...
> Sep  2 09:11:02 [kernel] lost page write due to I/O error on hdj1
> Sep  2 09:11:02 [kernel] end_request: I/O error, dev hdj, sector 586099263
> Sep  2 09:11:02 [kernel] md: disabled device hdj1, could not read superblock.
> Sep  2 09:11:02 [kernel] md: hdj1 has invalid sb, not importing!
> Sep  2 09:11:02 [kernel] md: md_import_device returned -22

Ah, yeah, that looks bad to me.  And since it is on a write, that
probably means the drive has also ran out of the sectors it has
reserved for automatically replacing failed ones.  You could probably
run some additional tests using smartctl, but I think it is probably
just time to get a replacement drive.

> Does all of this suggest a complete and total drive failure? :-)  I orginally built this md4 array
> as a Raid 5 from three of the cheepest drives that I could find (Maxtors).  Then a couple of
> months later, one drive died.  So I converted the remaining to into a raid 1.  Now this one seems
> to have died.  I guess I am learning my leason about buying cheep hardware.  Fortunately I have a
> back-up of everything on this array.

Yeah, probably.  I don't know why exactly, but it seems quite common
that when you buy several drives of the same make/model at the same
time (and thus, they probably come from the same lot), if one fails,
you can expect others to fail soon afterwards.  It seems like you are
better off buying from different manufacturers in this case.

-Richard
-- 
gentoo-user@gentoo.org mailing list