[gentoo-user] systemd fails to mount nfs4 mounts

[gentoo-user] systemd fails to mount nfs4 mounts

[Daniel Frey]

I discovered that after my update the other day (systemd is up to date 
as of Wednesday) that my remote nfs mounts are failing on startup.

(Note: as per my other thread, I haven't tried to disable ipv6 yet. I 
want to figure this out first.)

I use an IMSM raid with an initramfs provided by dracut.

I have also set:

  NetworkManager-wait-online.service 
                                loaded active exited    Network Manager 
Wait Online

to wait until networkmanager starts up, and it appears to be working.

When booting, something complains about the module sunrpc not being able 
to be loaded. However, everything I've selected relating to nfs is built 
directly into the kernel, so there shouldn't be external modules to 
begin with.

After logging in, dropping to the shell and manually mounting these nfs 
mounts work.

systemctl status:
● mnt-nas-Pictures.mount 
                                loaded failed failed 
/mnt/nas/Pictures

status:

# systemctl status mnt-nas-Pictures.mount
● mnt-nas-Pictures.mount - /mnt/nas/Pictures
    Loaded: loaded (/etc/fstab; generated; vendor preset: disabled)
    Active: failed (Result: timeout) since Fri 2017-10-13 14:51:08 PDT; 
17min ago
     Where: /mnt/nas/Pictures
      What: nas:/Pictures
      Docs: man:fstab(5)
            man:systemd-fstab-generator(8)
   Process: 781 ExecMount=/bin/mount nas:/Pictures /mnt/nas/Pictures -t 
nfs4 -o rw,defaults,_netdev,intr,bg (code=killed, signal=TERM)

Oct 13 14:49:37 myboringpc systemd[1]: Mounting /mnt/nas/Pictures...
Oct 13 14:51:08 myboringpc systemd[1]: mnt-nas-Pictures.mount: Mounting 
timed out. Stopping.
Oct 13 14:51:08 myboringpc systemd[1]: mnt-nas-Pictures.mount: Mount 
process exited, code=killed status=15
Oct 13 14:51:08 myboringpc systemd[1]: Failed to mount /mnt/nas/Pictures.
Oct 13 14:51:08 myboringpc systemd[1]: mnt-nas-Pictures.mount: Unit 
entered failed state.

So it is saying it was killed. When booting up, systemd sits waiting for 
a start job for two minutes. I rebooted and checked again, and it said 
it timed out. I can see it's waiting for Network Manager Wait Online, 
then it tries to mount the nfs shares.

I presume this means the network is not ready. As I mentioned above, 
logging in and manually mounting is working fine.

Anyone have any suggestions? Of course I didn't notice this until I 
tried to sync pictures from my phone...

Dan

[gentoo-user] Re: systemd fails to mount nfs4 mounts

[Daniel Frey]

On 10/13/2017 03:13 PM, Daniel Frey wrote:
> I discovered that after my update the other day (systemd is up to date 
> as of Wednesday) that my remote nfs mounts are failing on startup.
> 
> (Note: as per my other thread, I haven't tried to disable ipv6 yet. I 
> want to figure this out first.)
> 
> I use an IMSM raid with an initramfs provided by dracut.
> 
> I have also set:
> 
>   NetworkManager-wait-online.service                                
> loaded active exited    Network Manager Wait Online
> 
> to wait until networkmanager starts up, and it appears to be working.
> 
> When booting, something complains about the module sunrpc not being able 
> to be loaded. However, everything I've selected relating to nfs is built 
> directly into the kernel, so there shouldn't be external modules to 
> begin with.
> 
> After logging in, dropping to the shell and manually mounting these nfs 
> mounts work.
> 
> systemctl status:
> ● mnt-nas-Pictures.mount                                loaded failed 
> failed /mnt/nas/Pictures
> 
> status:
> 
> # systemctl status mnt-nas-Pictures.mount
> ● mnt-nas-Pictures.mount - /mnt/nas/Pictures
>     Loaded: loaded (/etc/fstab; generated; vendor preset: disabled)
>     Active: failed (Result: timeout) since Fri 2017-10-13 14:51:08 PDT; 
> 17min ago
>      Where: /mnt/nas/Pictures
>       What: nas:/Pictures
>       Docs: man:fstab(5)
>             man:systemd-fstab-generator(8)
>    Process: 781 ExecMount=/bin/mount nas:/Pictures /mnt/nas/Pictures -t 
> nfs4 -o rw,defaults,_netdev,intr,bg (code=killed, signal=TERM)
> 
> Oct 13 14:49:37 myboringpc systemd[1]: Mounting /mnt/nas/Pictures...
> Oct 13 14:51:08 myboringpc systemd[1]: mnt-nas-Pictures.mount: Mounting 
> timed out. Stopping.
> Oct 13 14:51:08 myboringpc systemd[1]: mnt-nas-Pictures.mount: Mount 
> process exited, code=killed status=15
> Oct 13 14:51:08 myboringpc systemd[1]: Failed to mount /mnt/nas/Pictures.
> Oct 13 14:51:08 myboringpc systemd[1]: mnt-nas-Pictures.mount: Unit 
> entered failed state.
> 
> So it is saying it was killed. When booting up, systemd sits waiting for 
> a start job for two minutes. I rebooted and checked again, and it said 
> it timed out. I can see it's waiting for Network Manager Wait Online, 
> then it tries to mount the nfs shares.
> 
> I presume this means the network is not ready. As I mentioned above, 
> logging in and manually mounting is working fine.
> 
> Anyone have any suggestions? Of course I didn't notice this until I 
> tried to sync pictures from my phone...
> 
> Dan

I'm still having this issue, anyone have any ideas? I can see that 
NetworkManager-Wait-Online finishes, and that the mounting starts 
immediately after, but I don't think the network is quite up yet, 
resulting an all nfs mounts to timeout.

The computer is using a static IP, so it shouldn't even be waiting for 
dhcp...

I booted into openrc, and they all mount properly. Does anyone have any 
idea even how to start troubleshooting this?

Dan

Re: [gentoo-user] Re: systemd fails to mount nfs4 mounts

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 588 bytes --]

> I'm still having this issue, anyone have any ideas? I can see that
> NetworkManager-Wait-Online finishes, and that the mounting starts
> immediately after, but I don't think the network is quite up yet, resulting
> an all nfs mounts to timeout.
>
> The computer is using a static IP, so it shouldn't even be waiting for
> dhcp...
>
> I booted into openrc, and they all mount properly. Does anyone have any
> idea even how to start troubleshooting this?
>

I had this issue through some versions of systemd - but its fixed now (i'm
on ~adm64). Version is systemd-235-r1.

Are you ~arch?

[-- Attachment #2: Type: text/html, Size: 964 bytes --]

Re: [gentoo-user] Re: systemd fails to mount nfs4 mounts

[Daniel Frey]

On 10/28/2017 05:18 PM, Adam Carter wrote:
> 
>     I'm still having this issue, anyone have any ideas? I can see that
>     NetworkManager-Wait-Online finishes, and that the mounting starts
>     immediately after, but I don't think the network is quite up yet,
>     resulting an all nfs mounts to timeout.
> 
>     The computer is using a static IP, so it shouldn't even be waiting
>     for dhcp...
> 
>     I booted into openrc, and they all mount properly. Does anyone have
>     any idea even how to start troubleshooting this?
> 
> 
> I had this issue through some versions of systemd - but its fixed now 
> (i'm on ~adm64). Version is systemd-235-r1.
> 
> Are you ~arch?

No, I run stable. Currently running:

[IP-] [  ] sys-apps/systemd-233-r4:0/2

Come to think of it, I'm going to look back and see if there was an 
update around the time I started having problems. Maybe there was a 
regression of some sort.

Dan

Re: [gentoo-user] Re: systemd fails to mount nfs4 mounts

[Daniel Frey]

On 10/29/2017 08:15 AM, Daniel Frey wrote:
> Come to think of it, I'm going to look back and see if there was an 
> update around the time I started having problems. Maybe there was a 
> regression of some sort.
> 

So I bought a large SSD, and cloned to it. I'm not stuck with IMSM any 
more, but systemd still doesn't mount properly. Now that I don't have to 
deal with mdadm, I went back to openrc, and all is will. Although... it 
did take me a while to get rid of networkmanager.

Dan

Re: [gentoo-user] Re: systemd fails to mount nfs4 mounts

[Wols Lists]

On 01/11/17 20:25, Daniel Frey wrote:
> On 10/29/2017 08:15 AM, Daniel Frey wrote:
>> Come to think of it, I'm going to look back and see if there was an
>> update around the time I started having problems. Maybe there was a
>> regression of some sort.
>>
> 
> So I bought a large SSD, and cloned to it. I'm not stuck with IMSM any
> more, but systemd still doesn't mount properly. Now that I don't have to
> deal with mdadm, I went back to openrc, and all is will. Although... it
> did take me a while to get rid of networkmanager.
> 
What's the problem with mdadm and openrc?

That said, I've just been troubleshooting a right pain in the neck with
SUSE, Windows, and systemd on my laptop.

ANY trouble with your hard drives, and systemd dumps you in the recovery
console.

Windows WON'T SHUT DOWN PROPERLY most of the time.

And something messed up /home.

Easy enough to fix, when I eventually found out the cause. Run fsck on
/dev/sda8. Re-configure windows to tell it "shut down does NOT mean
hibernate, damn you!", and finally reboot actually got me into SUSE proper.

But I wish they'd document - and fix!!! - how to get systemd to mount
drives properly!!!

Cheers,
Wol

Re: [gentoo-user] Re: systemd fails to mount nfs4 mounts

[Rich Freeman]

On Wed, Nov 1, 2017 at 2:12 PM, Wols Lists <antlists@youngman.org.uk> wrote:
>
> Windows WON'T SHUT DOWN PROPERLY most of the time.
>
> And something messed up /home.
>
> Easy enough to fix, when I eventually found out the cause. Run fsck on
> /dev/sda8. Re-configure windows to tell it "shut down does NOT mean
> hibernate, damn you!", and finally reboot actually got me into SUSE proper.
>
> But I wish they'd document - and fix!!! - how to get systemd to mount
> drives properly!!!
>

By "properly" it sounds like you want it to mount filesystems that
were not cleanly unmounted without user intervention, or ignore a
failure to do so?

I think I'll stick with the way it works now.

However, if you want it to boot without warnings if the drive won't
cleanly mount you can just add a nofail option to fstab for the
filesystem.  Then your system will just continue booting without the
filesystem mounted if the linux mount command wouldn't mount it with
the normal invocation.  And then you get to clean up after whatever
daemon goes writing stuff in the empty mountpoint.

I imagine systemd is dropping to a recovery console in these
situations because most sysadmins want to know when a filesystem is
not cleanly mounting, and continued operation in this state is
unpredictable.  I'm sure it could be overriden if you really wanted
to...

-- 
Rich

Re: [gentoo-user] Re: systemd fails to mount nfs4 mounts

[Daniel Frey]

On 11/01/2017 02:12 PM, Wols Lists wrote:
> What's the problem with mdadm and openrc?
> 

openrc terminates mdmon too early and so every time I rebooted this 
machine when it had a RAID it marked the array as dirty and rebuilt it. 
The PC was not usable while it was rebuilding, it was so dang slow, not 
to mention extra wear and tear on the drives. I had a workaround to this 
but it stopped working so I went to systemd, which helped that problem, 
but systemd gave me new problems... like the inability to wait for a 
network to be up before mounting remote nfs drives. openrc handles this 
just fine, I have an old Mint install and it too worked fine without 
intervention. I spent a couple weeks trying to figure out why systemd 
refused to wait for networkmanager to bring the network up, I gave up.

The thing is it used to work with systemd, then there was a couple 
updates and it stopped working. I unmasked a newer version with the same 
results.

The best I got was systemd would sometimes mount half the remote nfs 
mounts and timeout on the other half.

Dan