[gentoo-user] Troubleshooting mounting local filesystems

[gentoo-user] Troubleshooting mounting local filesystems

[Daniel Frey]

Some background:

A little while back I had a drive drop out of my hardware RAID. I don't 
think this has anything to do with the problem I'm having, but I thought 
I should mention it. The RAID health is fine and I can see there's not 
any delay in dmesg (the RAID array is detected as /dev/sdc).

On startup, I share the RAID, and hence I mount it locally under /mnt, 
then again under /nfs4exports.

Note: I'm using openrc.

For some reason, the entry under /mnt does not mount on startup. There's 
no error or any indication of anything going wrong during startup 
(nothing in dmesg or /var/log/messages regarding any sort of mount trouble.)

So what happens is nfs starts up but it's missing the one export. I have 
to stop nfs, unmount the entry under /nfs4exports, unmount the entry 
under /mnt, then mount /dev/sdc1 to /mnt, the mount the entry under 
/nfs4exports. After this, everything is mounted properly and I restart nfs.

I looked at the /etc/init.d/localmount script and it's supposed to spit 
out a message if something cannot mount but it does not report any error.

Is there any sort of logging I can enable to tell me exactly what's 
happening? Other local filesystems (total of three) all mount fine.

Dan

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

On Sun, Dec 31, 2017 at 2:22 PM, Daniel Frey <djqfrey@gmail.com> wrote:

> Some background:
>
> A little while back I had a drive drop out of my hardware RAID. I don't
> think this has anything to do with the problem I'm having, but I thought I
> should mention it. The RAID health is fine and I can see there's not any
> delay in dmesg (the RAID array is detected as /dev/sdc).
>
> On startup, I share the RAID, and hence I mount it locally under /mnt,
> then again under /nfs4exports.
>
> Note: I'm using openrc.
>
> For some reason, the entry under /mnt does not mount on startup. There's
> no error or any indication of anything going wrong during startup (nothing
> in dmesg or /var/log/messages regarding any sort of mount trouble.)
>
> So what happens is nfs starts up but it's missing the one export. I have
> to stop nfs, unmount the entry under /nfs4exports, unmount the entry under
> /mnt, then mount /dev/sdc1 to /mnt, the mount the entry under /nfs4exports.
> After this, everything is mounted properly and I restart nfs.
>
> I looked at the /etc/init.d/localmount script and it's supposed to spit
> out a message if something cannot mount but it does not report any error.
>
> Is there any sort of logging I can enable to tell me exactly what's
> happening? Other local filesystems (total of three) all mount fine.
>
>
What does the fstab entry look like?

Can you cut and paste the terminal session of the post boot fixes?

[-- Attachment #2: Type: text/html, Size: 1890 bytes --]

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Jalus Bilieyich]

Did you perform this action:

rc-update add localmount default

?

On 12/30/2017 09:22 PM, Daniel Frey wrote:
> Some background:
> 
> A little while back I had a drive drop out of my hardware RAID. I don't
> think this has anything to do with the problem I'm having, but I thought
> I should mention it. The RAID health is fine and I can see there's not
> any delay in dmesg (the RAID array is detected as /dev/sdc).
> 
> On startup, I share the RAID, and hence I mount it locally under /mnt,
> then again under /nfs4exports.
> 
> Note: I'm using openrc.
> 
> For some reason, the entry under /mnt does not mount on startup. There's
> no error or any indication of anything going wrong during startup
> (nothing in dmesg or /var/log/messages regarding any sort of mount
> trouble.)
> 
> So what happens is nfs starts up but it's missing the one export. I have
> to stop nfs, unmount the entry under /nfs4exports, unmount the entry
> under /mnt, then mount /dev/sdc1 to /mnt, the mount the entry under
> /nfs4exports. After this, everything is mounted properly and I restart nfs.
> 
> I looked at the /etc/init.d/localmount script and it's supposed to spit
> out a message if something cannot mount but it does not report any error.
> 
> Is there any sort of logging I can enable to tell me exactly what's
> happening? Other local filesystems (total of three) all mount fine.
> 
> Dan
> 

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Peter Humphrey]

On Sunday, 31 December 2017 14:12:22 GMT Jalus Bilieyich wrote:
> Did you perform this action:
> 
> rc-update add localmount default
> 
> ?

It's in the boot run-level here.

-- 
Regards,
Peter.

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Jalus Bilieyich]

I just noticed it's in /etc/init.d

Sorry for wasting your time.

On 12/31/2017 09:14 AM, Peter Humphrey wrote:
> On Sunday, 31 December 2017 14:12:22 GMT Jalus Bilieyich wrote:
>> Did you perform this action:
>>
>> rc-update add localmount default
>>
>> ?
> 
> It's in the boot run-level here.
> 

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Daniel Frey]

On 12/30/17 19:32, Adam Carter wrote:
> On Sun, Dec 31, 2017 at 2:22 PM, Daniel Frey <djqfrey@gmail.com 
> <mailto:djqfrey@gmail.com>> wrote:
> 
>     Some background:
> 
>     A little while back I had a drive drop out of my hardware RAID. I
>     don't think this has anything to do with the problem I'm having, but
>     I thought I should mention it. The RAID health is fine and I can see
>     there's not any delay in dmesg (the RAID array is detected as /dev/sdc).
> 
>     On startup, I share the RAID, and hence I mount it locally under
>     /mnt, then again under /nfs4exports.
> 
>     Note: I'm using openrc.
> 
>     For some reason, the entry under /mnt does not mount on startup.
>     There's no error or any indication of anything going wrong during
>     startup (nothing in dmesg or /var/log/messages regarding any sort of
>     mount trouble.)
> 
>     So what happens is nfs starts up but it's missing the one export. I
>     have to stop nfs, unmount the entry under /nfs4exports, unmount the
>     entry under /mnt, then mount /dev/sdc1 to /mnt, the mount the entry
>     under /nfs4exports. After this, everything is mounted properly and I
>     restart nfs.
> 
>     I looked at the /etc/init.d/localmount script and it's supposed to
>     spit out a message if something cannot mount but it does not report
>     any error.
> 
>     Is there any sort of logging I can enable to tell me exactly what's
>     happening? Other local filesystems (total of three) all mount fine.
> 
> What does the fstab entry look like?
> 

The fstab entry is just:

/dev/sdc1    /mnt/raid    ext4    noatime    0 0

The first thing I did was check localmount and it is in the boot 
runlevel (some snipped):

# rc-update show boot
	        ....
            localmount | boot
		....

Other filesystems mounted OK.

It does list as started:
# rc-status boot
Runlevel: boot
  localmount       [  started  ]


 > Can you cut and paste the terminal session of the post boot fixes?

What are you asking for here?

Part of the problem is I restart this machine so infrequently I usually 
forget about the mounting problems until I try to access it remotely.

Dan

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Daniel Frey]

On 12/31/17 06:12, Jalus Bilieyich wrote:
> Did you perform this action:
> 
> rc-update add localmount default
> 
> ?

On my machine (as per my other post) it's in the boot runlevel.

Dan

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

>
> > Can you cut and paste the terminal session of the post boot fixes?
>
> What are you asking for here?
>

Just fishing for more info because I cant think of any circumstance that
would cause the issue you're seeing.

If it were me i'd force an fsck on next reboot, then reboot. I just had a
look on my machines and it looks like the old fsck every N mounts is
disabled everywhere by default these days.

[-- Attachment #2: Type: text/html, Size: 722 bytes --]

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Daniel Frey]

On 01/01/18 01:14, Adam Carter wrote:
>     > Can you cut and paste the terminal session of the post boot fixes?
> 
>     What are you asking for here?
> 
> 
> Just fishing for more info because I cant think of any circumstance that 
> would cause the issue you're seeing.
> 
> If it were me i'd force an fsck on next reboot, then reboot. I just had 
> a look on my machines and it looks like the old fsck every N mounts is 
> disabled everywhere by default these days.
> 

That's a good idea, I stopped nfs, umounted and ran fsck, and it came 
back clean. I've remounted and restarted nfs for now.

I'll try restarting later.

Dan

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Daniel Frey]

On 12/31/17 08:28, Daniel Frey wrote:
> On 12/31/17 06:12, Jalus Bilieyich wrote:
>> Did you perform this action:
>>
>> rc-update add localmount default
>>
>> ?
> 
> On my machine (as per my other post) it's in the boot runlevel.
> 
> Dan
	
OK, after turning on some logging I figured out what's happening.

When localmount runs /dev/sdc has not been created yet. Presumably it is 
getting created after localmount runs.

I suspect that this is because there's a delay during kernel 
initialization that loads firmware for some TV tuners I have, and after 
this I can see that /dev/sdc is created.

So the question now is... is it safe to add a delay to localmount? Or is 
there a better way (like... can I tell the kernel to wait a few seconds 
before running init?)

As this is mounted twice (local access and nfs access) in fstab, it 
would be best to fix it before localmount runs.

Dan

Re: [gentoo-user] Troubleshooting mounting local filesystems

[Daniel Frey]

On 01/21/18 11:15, Daniel Frey wrote:
> On 12/31/17 08:28, Daniel Frey wrote:
>> On 12/31/17 06:12, Jalus Bilieyich wrote:
>>> Did you perform this action:
>>>
>>> rc-update add localmount default
>>>
>>> ?
>>
>> On my machine (as per my other post) it's in the boot runlevel.
>>
>> Dan
> 
> OK, after turning on some logging I figured out what's happening.
> 
> When localmount runs /dev/sdc has not been created yet. Presumably it is 
> getting created after localmount runs.
> 
> I suspect that this is because there's a delay during kernel 
> initialization that loads firmware for some TV tuners I have, and after 
> this I can see that /dev/sdc is created.
> 
> So the question now is... is it safe to add a delay to localmount? Or is 
> there a better way (like... can I tell the kernel to wait a few seconds 
> before running init?)
> 
> As this is mounted twice (local access and nfs access) in fstab, it 
> would be best to fix it before localmount runs.
> 
> Dan

OK, I found the rootdelay paramater but it didn't do what it needed 
(presumably because the root partition wasn't mounted so creating 
entries in /dev didn't happen any faster.)

I added the following three lines to add a delay:

         # Delay to allow devices to register
         einfo "Waiting for RAID devices to settle/register..."
         sleep 25

I've rebooted a few times and it starts up correctly now. I don't really 
care if that PC starts in two seconds, I need it to start up 
consistently as it's acting as a server.

Dan