[gentoo-server] 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards

[gentoo-server] 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards

[Jared Klett]

hello Gentoo server folks,

	I have a 2U server with a 3ware 9500-12MI controller. there are two  
80 GB drives in RAID-1 which servers as the boot device, and ten 400  
GB drives in RAID-5 which I want to have mounted as a single volume  
at /data.

	if I start fresh and create the filesystem after the system has  
booted, and mount it, all is well. however, when I reboot the system,  
the filesystem won't be mounted and I'll get an error.

	to start, I used GNU parted to partition the device, which is at / 
dev/sdb. here's a transcript of the session:

Using /dev/sdb
(parted) mklabel
New disk label type? gpt
(parted) print
Disk geometry for /dev/sdb: 0kB - 3600GB
Disk label type: gpt
Number  Start   End     Size    File system  Name                  Flags
(parted) mkpart
Partition type?  [primary]?
File system type?  [ext2]? reiserfs
Start? 0.0
End? 3600G
(parted) print
Disk geometry for /dev/sdb: 0kB - 3600GB
Disk label type: gpt
Number  Start   End     Size    File system  Name                  Flags
1       17kB    3600GB  3600GB
(parted) quit

	I then do a plain:

root ~ # mkreiserfs /dev/sdb1
mkreiserfs 3.6.19 (2003 www.namesys.com)
Guessing about desired format.. Kernel 2.6.17-gentoo-r4 is running.
Format 3.6 with standard journal
Count of blocks on the device: 878881520
Number of blocks consumed by mkreiserfs formatting process: 35033
Blocksize: 4096
Hash function used to sort names: "r5"
Journal Size 8193 blocks (first block 18)
Journal Max transaction length 1024
inode generation number: 0
UUID: 80864192-b2b1-4f20-ab16-a1242e5cfa1b
ATTENTION: YOU SHOULD REBOOT AFTER FDISK!
         ALL DATA WILL BE LOST ON '/dev/sdb1'!
Continue (y/n):y
Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..ok

root ~ # mount /dev/sdb1 /data
root ~ # df -h
[ ... snip ... ]
/dev/sdb1             3.3T   33M  3.3T   1% /data

	and all is well.

	however, when I update /etc/fstab and reboot the machine, the kernel  
loads and mounts the reiserfs filesystems on the boot volume, but  
then I get:

reiserfs_open: the reiserfs superblock cannot be found /dev/sdb1.
Failed to open the filesystem.

	here's a screenshot of the console:

http://www.project55.net/noloveatboot.png

	I hit Control-D and allow the system to finish booting. then I  
attempted this:

root ~ # mount /dev/sdb1 /data
mount: you must specify the filesystem type
root ~ # mount -t reiserfs /dev/sdb1 /data
mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
        missing codepage or other error
        In some cases useful info is found in syslog - try
        dmesg | tail  or so
root ~ # dmesg | tail
[ ... snip ...]
FAT: bogus number of reserved sectors
VFS: Can't find a valid FAT filesystem on dev sdb1.
ReiserFS: sdb1: warning: sh-2021: reiserfs_fill_super: can not find  
reiserfs on sdb1

	this message makes me a bit suspicious. I've tried creating the  
reiserfs filesystem from inside parted using mkfs, but I just get  
this, in spite of having reiserfsprogs emerged before I emerged parted:

No Implementation: Support for creating reiserfs file systems is not  
implemented yet.

	I'm running kernel version 2.6.17-gentoo-r4, with the following  
enabled:

Block layer  --->
  [*] Support for Large Block Devices
Partition Types  --->
  [*] Advanced partition selection
  ...
  [*] EFI GUID Partition support

	just for complete information overload, here is the status of the  
RAID unit as reported by the 3ware controller:

Status: OK
Capacity: 3.27 TB
Type: RAID 5
Stripe: 256kB

	any insights would be most welcome!

cheers,

- Jared

-- 
Move 'sig'. For great justice.

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards

[Karl Hiramoto]

Did you reboot after creating the partition?
Perhaps try fdisk instead of parted?

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards

[Jared Klett]

On Aug 24, 2006, at 1:20 PM, Karl Hiramoto wrote:

> Did you reboot after creating the partition?

	yes... even if I don't mount the partition, it's unusable after a  
reboot, ala:

root ~ # mount /dev/sdb1 /data
mount: you must specify the filesystem type

> Perhaps try fdisk instead of parted?

	unfortunately, fdisk doesn't support devices larger than 2 TB.

cheers,

- Jared

-- 
No, no, no, don't tug on that. You never know what it might be  
attached to.


-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards

[Robert Welz]

Jared Klett wrote:

> Partition type?  [primary]?
> File system type?  [ext2]? reiserfs
> Start? 0.0
> End? 3600G
> (parted) print
> Disk geometry for /dev/sdb: 0kB - 3600GB
> Disk label type: gpt
> Number  Start   End     Size    File system  Name                  Flags
> 1       17kB    3600GB  3600GB
> (parted) quit

Hi.
I think the "file system" parameter is missing. IMHO it must be 83 or 
"Linux".

There are rumors for Reiserfs not as reliable as it should be, maybe you 
should consider ext3 journaled, especially since you have such a big 
volume, so an eventual recovery is much faster.


But I don't want to start a flame war; thats what I've heard at "Linux 
Mailserver Conference" a while ago.

BTW: Are jou running an UPS with your 3ware controller? I've just 
fiddled out how not to corrupt my HDs with the 3ware Inc 
7xxx/8xxx-series PATA/RAID controller when the nut-ups software shuts 
down on power failure.

Robert
-- 
gentoo-server@gentoo.org mailing list

RE: [gentoo-server] 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards

[Jesse, Rich]

One thing to check is that the kernel mods are either compiled in the
kernel or loaded before mounting the fs.

Probably not it, but I thought I'd check the easiest solution...

Rich

-----Original Message-----
From: Jared Klett [mailto:jklett@project55.net] 
Sent: Thursday, August 24, 2006 1:19 PM
To: gentoo-server@lists.gentoo.org
Subject: Re: [gentoo-server] 3.3 TB reiserfs volume not mounting at
boot, and is unusable afterwards

On Aug 24, 2006, at 1:20 PM, Karl Hiramoto wrote:

> Did you reboot after creating the partition?

	yes... even if I don't mount the partition, it's unusable after
a  
reboot, ala:

root ~ # mount /dev/sdb1 /data
mount: you must specify the filesystem type

> Perhaps try fdisk instead of parted?

	unfortunately, fdisk doesn't support devices larger than 2 TB.

cheers,

- Jared

-- 
No, no, no, don't tug on that. You never know what it might be  
attached to.


-- 
gentoo-server@gentoo.org mailing list

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards

[Jared Klett]

On Aug 24, 2006, at 2:48 PM, Robert Welz wrote:

> Jared Klett wrote:
>
>> Partition type?  [primary]?
>> File system type?  [ext2]? reiserfs
>> Start? 0.0
>> End? 3600G
>> (parted) print
>> Disk geometry for /dev/sdb: 0kB - 3600GB
>> Disk label type: gpt
>> Number  Start   End     Size    File system  Name                   
>> Flags
>> 1       17kB    3600GB  3600GB
>> (parted) quit
>
> Hi.
> I think the "file system" parameter is missing.

	hi! yes, that isn't correct... it should be "reiserfs".

	it doesn't appear until after you run mkreiserfs. example:

Using /dev/sdb
(parted) p
Disk geometry for /dev/sdb: 0kB - 3600GB
Disk label type: gpt
Number  Start   End     Size    File system  Name                  Flags
1       17kB    3600GB  3600GB  reiserfs

> IMHO it must be 83 or "Linux".

	according to the parted manual, it can be one of the following:

ext2
fat16, fat32
hfs, hfs+, hfsx
linux-swap
NTFS
reiserfs
ufs

> BTW: Are jou running an UPS with your 3ware controller? I've just  
> fiddled out how not to corrupt my HDs with the 3ware Inc 7xxx/8xxx- 
> series PATA/RAID controller when the nut-ups software shuts down on  
> power failure.

	I don't have a UPS, but the server has a redundant power supply  
which is connected to two separate power circuits. so if one goes  
out, no big deal. in addition, the 9500-12MI controller has a battery- 
backup unit (BBU) installed to preserve the write cache in case of a  
power failure.

	if you're concerned about power failures when using 3ware cards, I  
recommend turning off the write cache, or using a BBU (costs about  
$100) if you're using a 95XX-series controller.

	unfortunately the 7-8xxx series controllers don't have BBU capability.

cheers,

- Jared


-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards

[Jared Klett]

On Aug 24, 2006, at 3:17 PM, Jesse, Rich wrote:

> One thing to check is that the kernel mods are either compiled in the
> kernel or loaded before mounting the fs.
> Probably not it, but I thought I'd check the easiest solution...

	hi Rich - good call, but unfortunately I compile everything in when  
I build a kernel, as a rule.

- J

> -----Original Message-----
> From: Jared Klett [mailto:jklett@project55.net]
> Sent: Thursday, August 24, 2006 1:19 PM
> To: gentoo-server@lists.gentoo.org
> Subject: Re: [gentoo-server] 3.3 TB reiserfs volume not mounting at
> boot, and is unusable afterwards
>
> On Aug 24, 2006, at 1:20 PM, Karl Hiramoto wrote:
>
>> Did you reboot after creating the partition?
>
> 	yes... even if I don't mount the partition, it's unusable after
> a
> reboot, ala:
>
> root ~ # mount /dev/sdb1 /data
> mount: you must specify the filesystem type
>
>> Perhaps try fdisk instead of parted?
>
> 	unfortunately, fdisk doesn't support devices larger than 2 TB.
>
> cheers,
>
> - Jared
>
> -- 
> No, no, no, don't tug on that. You never know what it might be
> attached to.
>
>
> -- 
> gentoo-server@gentoo.org mailing list
>
> -- 
> gentoo-server@gentoo.org mailing list
>

-- 
"Capitalism is the uneven distribution of wealth, and socialism the  
even distribution of poverty."
          - Winston S. Churchill

-- 
gentoo-server@gentoo.org mailing list

[gentoo-server] SOLVED (3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards)

[Jared Klett]

	Rich's response got me to thinking - I had seen a reference back  
when I was building a similar server. after some creative googling, I  
found it:

http://www.coraid.com/support/linux/contrib/chernow/gpt.html

	the key note is at the bottom of the page:

"Without this on, strange things happen. For instance, rebooting the  
machine caused the file system to no longer be mountable or the GPT  
table got corrupted."

	aha! exactly my problem! I rebooted the machine and watched the  
console carefully, and noted this:

* System.map not found unable to check symbols

	hmmmm. after some more googling, I found this discussion:

http://tinyurl.com/o57hm

	so I found the solution to making this message go away to be to copy  
System.map to /., which is mounted when the message pops up during  
the boot process, whereas /boot and /usr are not. then we edit the  
block at the end of /sbin/modules-update to be:

if [ -d "`depdir`" -a -f /proc/modules ]
then
         if [ -f /System.map ]; then
                 depmod -a -F /System.map ${KV}
         else
                 ewarn "System.map not found - unable to check symbols"
         fi
fi

	and viola, the boot process is happy once more.

	as an added bonus, this resolves the issue with the filesystem being  
usable after a reboot, and mounting at boot time. woohoo!

	thanks to all that replied. it always helps to bounce ideas off  
others to get the juices flowing.

cheers,

- Jared


On Aug 24, 2006, at 12:18 PM, Jared Klett wrote:

> hello Gentoo server folks,
>
> 	I have a 2U server with a 3ware 9500-12MI controller. there are  
> two 80 GB drives in RAID-1 which servers as the boot device, and  
> ten 400 GB drives in RAID-5 which I want to have mounted as a  
> single volume at /data.
>
> 	if I start fresh and create the filesystem after the system has  
> booted, and mount it, all is well. however, when I reboot the  
> system, the filesystem won't be mounted and I'll get an error.
>
> 	to start, I used GNU parted to partition the device, which is at / 
> dev/sdb. here's a transcript of the session:
>
> Using /dev/sdb
> (parted) mklabel
> New disk label type? gpt
> (parted) print
> Disk geometry for /dev/sdb: 0kB - 3600GB
> Disk label type: gpt
> Number  Start   End     Size    File system  Name                   
> Flags
> (parted) mkpart
> Partition type?  [primary]?
> File system type?  [ext2]? reiserfs
> Start? 0.0
> End? 3600G
> (parted) print
> Disk geometry for /dev/sdb: 0kB - 3600GB
> Disk label type: gpt
> Number  Start   End     Size    File system  Name                   
> Flags
> 1       17kB    3600GB  3600GB
> (parted) quit
>
> 	I then do a plain:
>
> root ~ # mkreiserfs /dev/sdb1
> mkreiserfs 3.6.19 (2003 www.namesys.com)
> Guessing about desired format.. Kernel 2.6.17-gentoo-r4 is running.
> Format 3.6 with standard journal
> Count of blocks on the device: 878881520
> Number of blocks consumed by mkreiserfs formatting process: 35033
> Blocksize: 4096
> Hash function used to sort names: "r5"
> Journal Size 8193 blocks (first block 18)
> Journal Max transaction length 1024
> inode generation number: 0
> UUID: 80864192-b2b1-4f20-ab16-a1242e5cfa1b
> ATTENTION: YOU SHOULD REBOOT AFTER FDISK!
>         ALL DATA WILL BE LOST ON '/dev/sdb1'!
> Continue (y/n):y
> Initializing journal - 0%....20%....40%....60%....80%....100%
> Syncing..ok
>
> root ~ # mount /dev/sdb1 /data
> root ~ # df -h
> [ ... snip ... ]
> /dev/sdb1             3.3T   33M  3.3T   1% /data
>
> 	and all is well.
>
> 	however, when I update /etc/fstab and reboot the machine, the  
> kernel loads and mounts the reiserfs filesystems on the boot  
> volume, but then I get:
>
> reiserfs_open: the reiserfs superblock cannot be found /dev/sdb1.
> Failed to open the filesystem.
>
> 	here's a screenshot of the console:
>
> http://www.project55.net/noloveatboot.png
>
> 	I hit Control-D and allow the system to finish booting. then I  
> attempted this:
>
> root ~ # mount /dev/sdb1 /data
> mount: you must specify the filesystem type
> root ~ # mount -t reiserfs /dev/sdb1 /data
> mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
>        missing codepage or other error
>        In some cases useful info is found in syslog - try
>        dmesg | tail  or so
> root ~ # dmesg | tail
> [ ... snip ...]
> FAT: bogus number of reserved sectors
> VFS: Can't find a valid FAT filesystem on dev sdb1.
> ReiserFS: sdb1: warning: sh-2021: reiserfs_fill_super: can not find  
> reiserfs on sdb1
>
> 	this message makes me a bit suspicious. I've tried creating the  
> reiserfs filesystem from inside parted using mkfs, but I just get  
> this, in spite of having reiserfsprogs emerged before I emerged  
> parted:
>
> No Implementation: Support for creating reiserfs file systems is  
> not implemented yet.
>
> 	I'm running kernel version 2.6.17-gentoo-r4, with the following  
> enabled:
>
> Block layer  --->
>  [*] Support for Large Block Devices
> Partition Types  --->
>  [*] Advanced partition selection
>  ...
>  [*] EFI GUID Partition support
>
> 	just for complete information overload, here is the status of the  
> RAID unit as reported by the 3ware controller:
>
> Status: OK
> Capacity: 3.27 TB
> Type: RAID 5
> Stripe: 256kB
>
> 	any insights would be most welcome!
>
> cheers,
>
> - Jared
>
> -- 
> Move 'sig'. For great justice.
>
> -- 
> gentoo-server@gentoo.org mailing list
>

-- 
It is very dark. You are likely to be eaten by a grue.

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards

[Robert Welz]

Jared Klett wrote:

> 
>> BTW: Are jou running an UPS with your 3ware controller? I've just 
>> fiddled out how not to corrupt my HDs with the 3ware Inc 
>> 7xxx/8xxx-series PATA/RAID controller when the nut-ups software shuts 
>> down on power failure.
> 
>     I don't have a UPS, but the server has a redundant power supply 
> which is connected to two separate power circuits. so if one goes out, 
> no big deal. in addition, the 9500-12MI controller has a battery-backup 
> unit (BBU) installed to preserve the write cache in case of a power 
> failure.
> 
>     if you're concerned about power failures when using 3ware cards, I 
> recommend turning off the write cache, or using a BBU (costs about $100) 
> if you're using a 95XX-series controller.
Yes. First flush at shutdown then immediately following a switch-off of 
the cache.

Did you get the webserver management software to run? I couldn't, it 
always crashed. But the console client works. Some scripting and polling 
every 30 minutes and it sends emails on alerts and failures. But the 
3ware webserver would be more convenient, at least a bit I think.

Greetings,
Robert.
-- 
gentoo-server@gentoo.org mailing list

[gentoo-server] 3ware controllers (was: 3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards)

[Jared Klett]

On Aug 24, 2006, at 4:51 PM, Robert Welz wrote:

> Jared Klett wrote:
>
>>> BTW: Are jou running an UPS with your 3ware controller? I've just  
>>> fiddled out how not to corrupt my HDs with the 3ware Inc 7xxx/ 
>>> 8xxx-series PATA/RAID controller when the nut-ups software shuts  
>>> down on power failure.
>>     I don't have a UPS, but the server has a redundant power  
>> supply which is connected to two separate power circuits. so if  
>> one goes out, no big deal. in addition, the 9500-12MI controller  
>> has a battery-backup unit (BBU) installed to preserve the write  
>> cache in case of a power failure.
>>     if you're concerned about power failures when using 3ware  
>> cards, I recommend turning off the write cache, or using a BBU  
>> (costs about $100) if you're using a 95XX-series controller.
> Yes. First flush at shutdown then immediately following a switch- 
> off of the cache.
>
> Did you get the webserver management software to run? I couldn't,  
> it always crashed. But the console client works. Some scripting and  
> polling every 30 minutes and it sends emails on alerts and  
> failures. But the 3ware webserver would be more convenient, at  
> least a bit I think.

	I, too, have had difficulty on older Gentoo systems with a 2.4.x  
kernel.

	the latest release of 3DM2 (9.3.0.4) seems to have corrected a lot  
of issues when run with 7-8xxx controllers. I have a few systems with  
8006-2 controllers and one with a 7506-4 that have 3DM2 running  
happily. but all those are running 2.6.x kernels... so I'm not sure  
if there are still issues with 3DM2 on 2.4.x kernels.

cheers,

- Jared

-- 
It's yet another in a long series of diversions in an attempt to  
avoid responsibility.


-- 
gentoo-server@gentoo.org mailing list

RE: [gentoo-server] SOLVED (3.3 TB reiserfs volume not mounting at boot, and is unusable afterwards)

[Jesse, Rich]

GJ!  Saving your solution for future reference... :)

Rich 

-----Original Message-----
From: Jared Klett [mailto:jklett@project55.net] 
Sent: Thursday, August 24, 2006 3:36 PM
To: gentoo-server@lists.gentoo.org
Subject: [gentoo-server] SOLVED (3.3 TB reiserfs volume not mounting at
boot, and is unusable afterwards)

	Rich's response got me to thinking - I had seen a reference back

when I was building a similar server. after some creative googling, I  
found it:

http://www.coraid.com/support/linux/contrib/chernow/gpt.html

	the key note is at the bottom of the page:

"Without this on, strange things happen. For instance, rebooting the  
machine caused the file system to no longer be mountable or the GPT  
table got corrupted."

	aha! exactly my problem! I rebooted the machine and watched the

console carefully, and noted this:

* System.map not found unable to check symbols

	hmmmm. after some more googling, I found this discussion:

http://tinyurl.com/o57hm

	so I found the solution to making this message go away to be to
copy  
System.map to /., which is mounted when the message pops up during  
the boot process, whereas /boot and /usr are not. then we edit the  
block at the end of /sbin/modules-update to be:

if [ -d "`depdir`" -a -f /proc/modules ]
then
         if [ -f /System.map ]; then
                 depmod -a -F /System.map ${KV}
         else
                 ewarn "System.map not found - unable to check symbols"
         fi
fi

	and viola, the boot process is happy once more.

	as an added bonus, this resolves the issue with the filesystem
being  
usable after a reboot, and mounting at boot time. woohoo!

	thanks to all that replied. it always helps to bounce ideas off

others to get the juices flowing.

cheers,

- Jared

[snip]

-- 
gentoo-server@gentoo.org mailing list