[gentoo-user] [~amd64] NFS server broken again :(

[gentoo-user] [~amd64] NFS server broken again :(

[walt]

Last night when I powered off my machines NFS was working perfectly.  Today
it's broken again for the nth time:

#systemctl status nfs-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib64/systemd/system/nfs-server.service; enabled)
   Active: failed (Result: exit-code) since Mon 2014-10-27 11:50:38 PDT; 25min ago
  Process: 896 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
  Process: 893 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
  Process: 939 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE)
  Process: 936 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 939 (code=exited, status=1/FAILURE)

Oct 27 11:50:38 a6 rpc.nfsd[939]: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
Oct 27 11:50:38 a6 rpc.nfsd[939]: rpc.nfsd: unable to set any sockets for nfsd
Oct 27 11:50:38 a6 systemd[1]: nfs-server.service: main process exited, code=exited, status=1/FAILURE
Oct 27 11:50:38 a6 systemd[1]: Failed to start NFS server and services.
Oct 27 11:50:38 a6 systemd[1]: Unit nfs-server.service entered failed state.

#rpc.nfsd -d
rpc.nfsd: Checking netconfig for visible protocols.
rpc.nfsd: Enabling inet udp.
rpc.nfsd: Enabling inet tcp.
rpc.nfsd: Enabling inet6 udp.
rpc.nfsd: Enabling inet6 tcp.
rpc.nfsd: knfsd is currently down
rpc.nfsd: Writing version string to kernel: -2 +3 +4 
rpc.nfsd: Creating inet TCP socket.
rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
rpc.nfsd: Creating inet6 TCP socket.
rpc.nfsd: writing fd to kernel failed: errno 97 (Address family not supported by protocol)
rpc.nfsd: unable to set any sockets for nfsd

Same kernel (3.16.6-gentoo) and nfs-utils-1.3.0-r1 I've been using for weeks
with no trouble, so why today?

Packages updated yesterday have (I think) nothing to do with nfs:

#qlop -l |grep 'Oct 26'
Fri Oct 26 03:07:58 2012 >>> app-text/calibre-0.9.4
Fri Oct 26 03:08:13 2012 >>> app-mobilephone/obexd-0.46
Fri Oct 26 13:41:39 2012 >>> sys-fs/fuse-2.9.1-r1
Fri Oct 26 13:41:47 2012 >>> sys-fs/mtpfs-1.1
Sat Oct 26 15:34:57 2013 >>> media-video/avidemux-2.5.6-r2
Sun Oct 26 07:09:51 2014 >>> sys-libs/timezone-data-2014i-r1
Sun Oct 26 07:10:10 2014 >>> dev-python/py-1.4.26
Sun Oct 26 07:10:25 2014 >>> dev-python/pytest-2.6.4
Sun Oct 26 07:14:05 2014 >>> dev-lang/perl-5.20.1-r2
Sun Oct 26 07:14:28 2014 >>> dev-perl/DBI-1.631.0
Sun Oct 26 07:14:49 2014 >>> net-libs/polarssl-1.3.9
Sun Oct 26 07:15:25 2014 >>> dev-python/pillow-2.5.3-r1
Sun Oct 26 08:41:38 2014 >>> net-libs/webkit-gtk-2.4.7

Any ideas on how to debug this?

Thanks.

Re: [gentoo-user] [~amd64] NFS server broken again :(

[Canek Peláez Valdés]

On Mon, Oct 27, 2014 at 1:38 PM, walt <w41ter@gmail.com> wrote:
> Last night when I powered off my machines NFS was working perfectly.  Today
> it's broken again for the nth time:
>
> #systemctl status nfs-server
> ● nfs-server.service - NFS server and services
>    Loaded: loaded (/usr/lib64/systemd/system/nfs-server.service; enabled)
>    Active: failed (Result: exit-code) since Mon 2014-10-27 11:50:38 PDT; 25min ago
>   Process: 896 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
>   Process: 893 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
>   Process: 939 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE)
>   Process: 936 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
>  Main PID: 939 (code=exited, status=1/FAILURE)
>
> Oct 27 11:50:38 a6 rpc.nfsd[939]: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
> Oct 27 11:50:38 a6 rpc.nfsd[939]: rpc.nfsd: unable to set any sockets for nfsd
> Oct 27 11:50:38 a6 systemd[1]: nfs-server.service: main process exited, code=exited, status=1/FAILURE
> Oct 27 11:50:38 a6 systemd[1]: Failed to start NFS server and services.
> Oct 27 11:50:38 a6 systemd[1]: Unit nfs-server.service entered failed state.
>
> #rpc.nfsd -d
> rpc.nfsd: Checking netconfig for visible protocols.
> rpc.nfsd: Enabling inet udp.
> rpc.nfsd: Enabling inet tcp.
> rpc.nfsd: Enabling inet6 udp.
> rpc.nfsd: Enabling inet6 tcp.
> rpc.nfsd: knfsd is currently down
> rpc.nfsd: Writing version string to kernel: -2 +3 +4
> rpc.nfsd: Creating inet TCP socket.
> rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
> rpc.nfsd: Creating inet6 TCP socket.
> rpc.nfsd: writing fd to kernel failed: errno 97 (Address family not supported by protocol)
> rpc.nfsd: unable to set any sockets for nfsd
>
> Same kernel (3.16.6-gentoo) and nfs-utils-1.3.0-r1 I've been using for weeks
> with no trouble, so why today?
>
> Packages updated yesterday have (I think) nothing to do with nfs:
>
> #qlop -l |grep 'Oct 26'
> Fri Oct 26 03:07:58 2012 >>> app-text/calibre-0.9.4
> Fri Oct 26 03:08:13 2012 >>> app-mobilephone/obexd-0.46
> Fri Oct 26 13:41:39 2012 >>> sys-fs/fuse-2.9.1-r1
> Fri Oct 26 13:41:47 2012 >>> sys-fs/mtpfs-1.1
> Sat Oct 26 15:34:57 2013 >>> media-video/avidemux-2.5.6-r2
> Sun Oct 26 07:09:51 2014 >>> sys-libs/timezone-data-2014i-r1
> Sun Oct 26 07:10:10 2014 >>> dev-python/py-1.4.26
> Sun Oct 26 07:10:25 2014 >>> dev-python/pytest-2.6.4
> Sun Oct 26 07:14:05 2014 >>> dev-lang/perl-5.20.1-r2
> Sun Oct 26 07:14:28 2014 >>> dev-perl/DBI-1.631.0
> Sun Oct 26 07:14:49 2014 >>> net-libs/polarssl-1.3.9
> Sun Oct 26 07:15:25 2014 >>> dev-python/pillow-2.5.3-r1
> Sun Oct 26 08:41:38 2014 >>> net-libs/webkit-gtk-2.4.7
>
> Any ideas on how to debug this?

I think I know the answer. Some days ago you moved /etc/conf.d for
NetworkManager to work, right? Where does the environment variable
RPCNFSDARGS is defined? I'm willing to bet that is in a /etc/conf.d
file.

Could you please post here the contents of nfs-server.service, and if
it exists, the files inside /etc/systemd/system/nfs-server.service.d
and their contents?

Regards.
-- 
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[gentoo-user] Re: [~amd64] NFS server broken again :(

[walt]

On 10/27/2014 12:56 PM, Canek Peláez Valdés wrote:
> On Mon, Oct 27, 2014 at 1:38 PM, walt <w41ter@gmail.com> wrote:
>> Last night when I powered off my machines NFS was working perfectly.  Today
>> it's broken again for the nth time:
>>
>> #systemctl status nfs-server
>> ● nfs-server.service - NFS server and services
>>    Loaded: loaded (/usr/lib64/systemd/system/nfs-server.service; enabled)
>>    Active: failed (Result: exit-code) since Mon 2014-10-27 11:50:38 PDT; 25min ago
>>   Process: 896 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
>>   Process: 893 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
>>   Process: 939 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE)
 
> I think I know the answer. Some days ago you moved /etc/conf.d for
> NetworkManager to work, right? Where does the environment variable
> RPCNFSDARGS is defined? I'm willing to bet that is in a /etc/conf.d
> file.
> 
> Could you please post here the contents of nfs-server.service, and if
> it exists, the files inside /etc/systemd/system/nfs-server.service.d
> and their contents?

Bingo again :)  Your question led me to the answer, which I think is a bug
in /usr/lib64/systemd/system/nfs-server.service.

Here's why the bug showed up just this morning:  way back at the beginning
of systemd I stole some .service files from RedHat Fedora, including one
named 'nfs.service'.

Turns out the foreign RedHat file was starting rpcbind for me all those
months and, when I deleted it last night, rpcbind didn't get started this
morning by nfs-server.service from gentoo (which I think is a bug).

#cat nfs-server.service 
[Unit]
Description=NFS server and services
Requires= network.target proc-fs-nfsd.mount rpcbind.target
Requires= nfs-mountd.service
Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service
Wants=rpc-statd-notify.service

After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
After= nfs-idmapd.service rpc-statd.service
After= rpc-gssd.service rpc-svcgssd.service
Before= rpc-statd-notify.service

[Service]
EnvironmentFile=/etc/conf.d/nfs

Type=oneshot
RemainAfterExit=yes
ExecStartPre=/usr/sbin/exportfs -r
ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
ExecStop=/usr/sbin/rpc.nfsd 0
ExecStopPost=/usr/sbin/exportfs -au
ExecStopPost=/usr/sbin/exportfs -f

ExecReload=/usr/sbin/exportfs -r

[Install]
WantedBy=multi-user.target


I can see that rpcbind.target is Required, but NOT rpcbind.service. AFAICT
rpc.target does nothing (please explain if I'm wrong about that).


BTW, /etc/conf.d/nfs doesn't define RPCNFSDARGS because it is intended
for use by openrc (another bug?):

#cat /etc/conf.d/nfs
# /etc/conf.d/nfs

# If you wish to set the port numbers for lockd,
# please see /etc/sysctl.conf

# Optional services to include in default `/etc/init.d/nfs start`
# For NFSv4 users, you'll want to add "rpc.idmapd" here.
NFS_NEEDED_SERVICES="rpc.idmapd"

# Number of servers to be started up by default
OPTS_RPC_NFSD="8"

# Options to pass to rpc.mountd
# ex. OPTS_RPC_MOUNTD="-p 32767"
OPTS_RPC_MOUNTD=""

# Options to pass to rpc.statd
# ex. OPTS_RPC_STATD="-p 32765 -o 32766"
OPTS_RPC_STATD=""

# Options to pass to rpc.idmapd
OPTS_RPC_IDMAPD=""

# Options to pass to rpc.gssd
OPTS_RPC_GSSD=""

# Options to pass to rpc.svcgssd
OPTS_RPC_SVCGSSD=""

# Options to pass to rpc.rquotad (requires sys-fs/quota)
OPTS_RPC_RQUOTAD=""

# Timeout (in seconds) for exportfs
EXPORTFS_TIMEOUT=30

# Options to set in the nfsd filesystem (/proc/fs/nfsd/).
# Format is <option>=<value>.  Multiple options are allowed.
#OPTS_NFSD="nfsv4leasetime=30 max_block_size=4096"


Thanks Canek!

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Canek Peláez Valdés]

On Mon, Oct 27, 2014 at 5:46 PM, walt <w41ter@gmail.com> wrote:
> On 10/27/2014 12:56 PM, Canek Peláez Valdés wrote:
>> On Mon, Oct 27, 2014 at 1:38 PM, walt <w41ter@gmail.com> wrote:
>>> Last night when I powered off my machines NFS was working perfectly.  Today
>>> it's broken again for the nth time:
>>>
>>> #systemctl status nfs-server
>>> ● nfs-server.service - NFS server and services
>>>    Loaded: loaded (/usr/lib64/systemd/system/nfs-server.service; enabled)
>>>    Active: failed (Result: exit-code) since Mon 2014-10-27 11:50:38 PDT; 25min ago
>>>   Process: 896 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
>>>   Process: 893 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
>>>   Process: 939 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE)
>
>> I think I know the answer. Some days ago you moved /etc/conf.d for
>> NetworkManager to work, right? Where does the environment variable
>> RPCNFSDARGS is defined? I'm willing to bet that is in a /etc/conf.d
>> file.
>>
>> Could you please post here the contents of nfs-server.service, and if
>> it exists, the files inside /etc/systemd/system/nfs-server.service.d
>> and their contents?
>
> Bingo again :)  Your question led me to the answer, which I think is a bug
> in /usr/lib64/systemd/system/nfs-server.service.
>
> Here's why the bug showed up just this morning:  way back at the beginning
> of systemd I stole some .service files from RedHat Fedora, including one
> named 'nfs.service'.

Gentoo nfs-utils package still doesn't include a systemd unit file,
really? With a perfunctory look at
/usr/portage/net-fs/nfs-utils/files, I see several *.service files. I
highly recommend using the unit files provided by the Gentoo devs.

> Turns out the foreign RedHat file was starting rpcbind for me all those
> months and, when I deleted it last night, rpcbind didn't get started this
> morning by nfs-server.service from gentoo (which I think is a bug).
>
> #cat nfs-server.service
> [Unit]
> Description=NFS server and services
> Requires= network.target proc-fs-nfsd.mount rpcbind.target
> Requires= nfs-mountd.service
> Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service
> Wants=rpc-statd-notify.service
>
> After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
> After= nfs-idmapd.service rpc-statd.service
> After= rpc-gssd.service rpc-svcgssd.service
> Before= rpc-statd-notify.service
>
> [Service]
> EnvironmentFile=/etc/conf.d/nfs
>
> Type=oneshot
> RemainAfterExit=yes
> ExecStartPre=/usr/sbin/exportfs -r
> ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
> ExecStop=/usr/sbin/rpc.nfsd 0
> ExecStopPost=/usr/sbin/exportfs -au
> ExecStopPost=/usr/sbin/exportfs -f
>
> ExecReload=/usr/sbin/exportfs -r
>
> [Install]
> WantedBy=multi-user.target
>
>
> I can see that rpcbind.target is Required, but NOT rpcbind.service. AFAICT
> rpc.target does nothing (please explain if I'm wrong about that).

I have no idea; I haven't set an NFS server in years. However, most
target units usually are kinda "virtuals"; the bring other units up.

> BTW, /etc/conf.d/nfs doesn't define RPCNFSDARGS because it is intended
> for use by openrc (another bug?):

Something should define RPCNFSDARGS; perhaps a drop-in?

> #cat /etc/conf.d/nfs
> # /etc/conf.d/nfs
>
> # If you wish to set the port numbers for lockd,
> # please see /etc/sysctl.conf
>
> # Optional services to include in default `/etc/init.d/nfs start`
> # For NFSv4 users, you'll want to add "rpc.idmapd" here.
> NFS_NEEDED_SERVICES="rpc.idmapd"
>
> # Number of servers to be started up by default
> OPTS_RPC_NFSD="8"
>
> # Options to pass to rpc.mountd
> # ex. OPTS_RPC_MOUNTD="-p 32767"
> OPTS_RPC_MOUNTD=""
>
> # Options to pass to rpc.statd
> # ex. OPTS_RPC_STATD="-p 32765 -o 32766"
> OPTS_RPC_STATD=""
>
> # Options to pass to rpc.idmapd
> OPTS_RPC_IDMAPD=""
>
> # Options to pass to rpc.gssd
> OPTS_RPC_GSSD=""
>
> # Options to pass to rpc.svcgssd
> OPTS_RPC_SVCGSSD=""
>
> # Options to pass to rpc.rquotad (requires sys-fs/quota)
> OPTS_RPC_RQUOTAD=""
>
> # Timeout (in seconds) for exportfs
> EXPORTFS_TIMEOUT=30
>
> # Options to set in the nfsd filesystem (/proc/fs/nfsd/).
> # Format is <option>=<value>.  Multiple options are allowed.
> #OPTS_NFSD="nfsv4leasetime=30 max_block_size=4096"
>
>
> Thanks Canek!

Walt, from time to time run "systemd-delta" and see what
configurations of yours differ from upstream (either systemd and/or
Gentoo). I think you should be using nfs-utils' included unit files.

Regards.
-- 
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Tom H]

On Mon, Oct 27, 2014 at 7:46 PM, walt <w41ter@gmail.com> wrote:
> On 10/27/2014 12:56 PM, Canek Peláez Valdés wrote:
>> On Mon, Oct 27, 2014 at 1:38 PM, walt <w41ter@gmail.com> wrote:
>>>
>>> Last night when I powered off my machines NFS was working perfectly.  Today
>>> it's broken again for the nth time:
>>>
>>> #systemctl status nfs-server
>>> ● nfs-server.service - NFS server and services
>>>    Loaded: loaded (/usr/lib64/systemd/system/nfs-server.service; enabled)
>>>    Active: failed (Result: exit-code) since Mon 2014-10-27 11:50:38 PDT; 25min ago
>>>   Process: 896 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
>>>   Process: 893 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
>>>   Process: 939 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE)
>>
>> I think I know the answer. Some days ago you moved /etc/conf.d for
>> NetworkManager to work, right? Where does the environment variable
>> RPCNFSDARGS is defined? I'm willing to bet that is in a /etc/conf.d
>> file.
>>
>> Could you please post here the contents of nfs-server.service, and if
>> it exists, the files inside /etc/systemd/system/nfs-server.service.d
>> and their contents?
>
> Bingo again :)  Your question led me to the answer, which I think is a bug
> in /usr/lib64/systemd/system/nfs-server.service.
>
> Here's why the bug showed up just this morning:  way back at the beginning
> of systemd I stole some .service files from RedHat Fedora, including one
> named 'nfs.service'.

Both RH and Gentoo have now adopted the upstream systemd units, which
don't include "nfs.service". But RH creates an "nfs.service" alias to
"nfs-server.service" so as not to upset RHEL sysadmin muscle memory
and scripts.


> Turns out the foreign RedHat file was starting rpcbind for me all those
> months and, when I deleted it last night, rpcbind didn't get started this
> morning by nfs-server.service from gentoo (which I think is a bug).

Does rpcbind.target exist? Does rpcbind.service have a "Requires" or
"Wants" for rpcbind.target? Is rpcbind.service enabled?


> #cat nfs-server.service
> [Unit]
> Description=NFS server and services
> Requires= network.target proc-fs-nfsd.mount rpcbind.target
> Requires= nfs-mountd.service
> Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service
> Wants=rpc-statd-notify.service
>
> After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
> After= nfs-idmapd.service rpc-statd.service
> After= rpc-gssd.service rpc-svcgssd.service
> Before= rpc-statd-notify.service
>
> [Service]
> EnvironmentFile=/etc/conf.d/nfs
>
> Type=oneshot
> RemainAfterExit=yes
> ExecStartPre=/usr/sbin/exportfs -r
> ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS
> ExecStop=/usr/sbin/rpc.nfsd 0
> ExecStopPost=/usr/sbin/exportfs -au
> ExecStopPost=/usr/sbin/exportfs -f
>
> ExecReload=/usr/sbin/exportfs -r
>
> [Install]
> WantedBy=multi-user.target
>
> I can see that rpcbind.target is Required, but NOT rpcbind.service. AFAICT
> rpc.target does nothing (please explain if I'm wrong about that).
>
>
> BTW, /etc/conf.d/nfs doesn't define RPCNFSDARGS because it is intended
> for use by openrc (another bug?):

I don't have access to a Gentoo box with nfs at the moment in order to
check this but IIRC Gentoo used to use OPTS_RPC_NFSD, OPTS_RPC_MOUNTD,
OPTS_RPC_STATD but it's now using upstream's RPCNFSDARGS,
RPCMOUNTDARGS, STATDARGS, at least in its systemd units. Again IIRC
the ebuild only changes the upstream "EnvironmentFile=" value and
deep-sixes nfs-config.service.


> #cat /etc/conf.d/nfs
> # /etc/conf.d/nfs
>
> # If you wish to set the port numbers for lockd,
> # please see /etc/sysctl.conf
>
> # Optional services to include in default `/etc/init.d/nfs start`
> # For NFSv4 users, you'll want to add "rpc.idmapd" here.
> NFS_NEEDED_SERVICES="rpc.idmapd"
>
> # Number of servers to be started up by default
> OPTS_RPC_NFSD="8"
>
> # Options to pass to rpc.mountd
> # ex. OPTS_RPC_MOUNTD="-p 32767"
> OPTS_RPC_MOUNTD=""
>
> # Options to pass to rpc.statd
> # ex. OPTS_RPC_STATD="-p 32765 -o 32766"
> OPTS_RPC_STATD=""
>
> # Options to pass to rpc.idmapd
> OPTS_RPC_IDMAPD=""
>
> # Options to pass to rpc.gssd
> OPTS_RPC_GSSD=""
>
> # Options to pass to rpc.svcgssd
> OPTS_RPC_SVCGSSD=""
>
> # Options to pass to rpc.rquotad (requires sys-fs/quota)
> OPTS_RPC_RQUOTAD=""
>
> # Timeout (in seconds) for exportfs
> EXPORTFS_TIMEOUT=30
>
> # Options to set in the nfsd filesystem (/proc/fs/nfsd/).
> # Format is <option>=<value>.  Multiple options are allowed.
> #OPTS_NFSD="nfsv4leasetime=30 max_block_size=4096"

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Tom H]

On Mon, Oct 27, 2014 at 10:49 PM, Tom H <tomh0665@gmail.com> wrote:

> Does rpcbind.target exist? Does rpcbind.service have a "Requires" or
> "Wants" for rpcbind.target? Is rpcbind.service enabled?

...

> I don't have access to a Gentoo box with nfs at the moment in order to
> check this but IIRC Gentoo used to use OPTS_RPC_NFSD, OPTS_RPC_MOUNTD,
> OPTS_RPC_STATD but it's now using upstream's RPCNFSDARGS,
> RPCMOUNTDARGS, STATDARGS, at least in its systemd units. Again IIRC
> the ebuild only changes the upstream "EnvironmentFile=" value and
> deep-sixes nfs-config.service.

I've just had the unoriginal idea of looking at my portage tree's
nfs-utils and rpcbind ebuilds and files...

The 1.2.9 nfs-utils ebuild has "systemd_dounit
"${FILESDIR}"/nfsd.service" where nfsd.service has
"Requires=rpcbind.service" and "After=rpcbind.service".

The 1.3.0 nfs-utils ebuild has "systemd_dounit
systemd/*.{mount,service,target}" where nfs-server.service has
"Requires=rpcbind.target" and After=rpcbind.target".

The rpcbind ebuild has "systemd_dounit "${FILESDIR}"/${PN}.service" so
there's no "${PN}.target" (and there isn't a ".target" unit in
$FILESDIR).

And, I was wrong:

cat $FILESDIR/nfs.confd
# /etc/conf.d/nfs

# If you wish to set the port numbers for lockd,
# please see /etc/sysctl.conf

# Optional services to include in default `/etc/init.d/nfs start`
# For NFSv4 users, you'll want to add "rpc.idmapd" here.
NFS_NEEDED_SERVICES=""

# Number of servers to be started up by default
OPTS_RPC_NFSD="8"

# Options to pass to rpc.mountd
# ex. OPTS_RPC_MOUNTD="-p 32767"
OPTS_RPC_MOUNTD=""

# Options to pass to rpc.statd
# ex. OPTS_RPC_STATD="-p 32765 -o 32766"
OPTS_RPC_STATD=""

# Options to pass to rpc.idmapd
OPTS_RPC_IDMAPD=""

# Options to pass to rpc.gssd
OPTS_RPC_GSSD=""

# Options to pass to rpc.svcgssd
OPTS_RPC_SVCGSSD=""

# Options to pass to rpc.rquotad (requires sys-fs/quota)
OPTS_RPC_RQUOTAD=""

# Timeout (in seconds) for exportfs
EXPORTFS_TIMEOUT=30

# Options to set in the nfsd filesystem (/proc/fs/nfsd/).
# Format is <option>=<value>.  Multiple options are allowed.
#OPTS_NFSD="nfsv4leasetime=30 max_block_size=4096"

so if you're using the 1.3.0 nfs-utils ebuild, you have to set up the
variables yourself, if you need to pass different values to the
daemons.

[gentoo-user] Re: [~amd64] NFS server broken again :(

[walt]

On 10/27/2014 08:22 PM, Tom H wrote:
> The 1.2.9 nfs-utils ebuild has "systemd_dounit
> "${FILESDIR}"/nfsd.service" where nfsd.service has
> "Requires=rpcbind.service" and "After=rpcbind.service".
> 
> The 1.3.0 nfs-utils ebuild has "systemd_dounit
> systemd/*.{mount,service,target}" where nfs-server.service has
> "Requires=rpcbind.target" and After=rpcbind.target".

Yes, and that little change caused the breakage that inspired this
thread in the first place :)

I have a Fedora 20 machine running in VirtualBox and I see they've
already fixed the same breakage by going back to 'rpcbind.service'
in their nfs-server.service file.

I see they also define all those $RPCFOO variables in /etc/sysconfig/nfs,
which are mostly null-strings anyway, which is why my nfs server is
working correctly without those variables.

(Working correctly *after* "systemctl enable rpcbind", that is.)

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Tom H]

On Tue, Oct 28, 2014 at 6:18 PM, walt <w41ter@gmail.com> wrote:
> On 10/27/2014 08:22 PM, Tom H wrote:
>>
>> The 1.2.9 nfs-utils ebuild has "systemd_dounit
>> "${FILESDIR}"/nfsd.service" where nfsd.service has
>> "Requires=rpcbind.service" and "After=rpcbind.service".
>>
>> The 1.3.0 nfs-utils ebuild has "systemd_dounit
>> systemd/*.{mount,service,target}" where nfs-server.service has
>> "Requires=rpcbind.target" and After=rpcbind.target".
>
> Yes, and that little change caused the breakage that inspired this
> thread in the first place :)
>
> I have a Fedora 20 machine running in VirtualBox and I see they've
> already fixed the same breakage by going back to 'rpcbind.service'
> in their nfs-server.service file.
>
> I see they also define all those $RPCFOO variables in /etc/sysconfig/nfs,
> which are mostly null-strings anyway, which is why my nfs server is
> working correctly without those variables.
>
> (Working correctly *after* "systemctl enable rpcbind", that is.)

"systemctl enable rpcbind" or "systemctl start rpcbind"? (Or
"systemctl enable rpcbind" and reboot?)

I've been wondering about "rpcbind.target". I'm not using systemd on
Gentoo but on my laptop running Ubuntu 15.04 (yes, 15 with systemd
215), "/lib/systemd/system/rpcbind.target" is provided by systemd.
It's also part of the upstream tarball so it must be part of the
Gentoo package - and, I have to assume, part of the Fedora systemd
package since Lennart's its maintainer there.

Since Gentoo's rpcbind.service has "Wants=rpcbind.target" and
"Before=rpcbind.target"", having nfs-server.service depend on
rpcbind.target rather than rpcbind.service should work as long as
rpcbind.service is enabled.

But having "Requires=rpcbind.service" and "After=rpcbind.service",
like nfsd.service has/had, means that you don't have to enable
rpcbind.service.

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Rich Freeman]

On Tue, Oct 28, 2014 at 9:36 PM, Tom H <tomh0665@gmail.com> wrote:
> Since Gentoo's rpcbind.service has "Wants=rpcbind.target" and
> "Before=rpcbind.target"", having nfs-server.service depend on
> rpcbind.target rather than rpcbind.service should work as long as
> rpcbind.service is enabled.
>
> But having "Requires=rpcbind.service" and "After=rpcbind.service",
> like nfsd.service has/had, means that you don't have to enable
> rpcbind.service.
>

I was just looking at that and thinking the same thing.  Nothing is
really forcing rpcbind to load the way things are specified right now.
If a service really requires another service to operate, it should say
that.  There is no problem doing that via a target, but then the
target still needs to pull it in.

There seems a general tendency in systemd to express dependencies as
"after" instead of "requires."  That is fine if the service doesn't
really require something else, but if there really is a true
dependency then it just causes problems when somebody doesn't notice
and fails to enable the other unit.

--
Rich

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Tom H]

On Tue, Oct 28, 2014 at 9:50 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Tue, Oct 28, 2014 at 9:36 PM, Tom H <tomh0665@gmail.com> wrote:
>> Since Gentoo's rpcbind.service has "Wants=rpcbind.target" and
>> "Before=rpcbind.target"", having nfs-server.service depend on
>> rpcbind.target rather than rpcbind.service should work as long as
>> rpcbind.service is enabled.
>>
>> But having "Requires=rpcbind.service" and "After=rpcbind.service",
>> like nfsd.service has/had, means that you don't have to enable
>> rpcbind.service.
>
> I was just looking at that and thinking the same thing.  Nothing is
> really forcing rpcbind to load the way things are specified right now.
> If a service really requires another service to operate, it should say
> that.  There is no problem doing that via a target, but then the
> target still needs to pull it in.

Wouldn't the solution to this problem to have a news item to let the
user know that rpcbind was being started as a dependency of
nfsd.service but that it now needs to be enabled in order to be
started by nfs-server.service?


> There seems a general tendency in systemd to express dependencies as
> "after" instead of "requires."  That is fine if the service doesn't
> really require something else, but if there really is a true
> dependency then it just causes problems when somebody doesn't notice
> and fails to enable the other unit.

AFAIK they're completely different and you can have service1 have a
"Requires" on service2 but have service2 start before service1. So if
someone's using "After" and expecting "Requires", he/she is bound to
be surprised by the result.

Is "After" really necessary as an option? I've never come across a
service that uses "After" without a "Requires" or a Wants" but I've
never taken the time to look.

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Rich Freeman]

On Fri, Oct 31, 2014 at 1:34 PM, Tom H <tomh0665@gmail.com> wrote:
> Is "After" really necessary as an option? I've never come across a
> service that uses "After" without a "Requires" or a Wants" but I've
> never taken the time to look.
>

Hmm, I found After more common that Wants, but maybe I only look at
units that have problems.  :)

I think the intent is to handle optional dependencies, but in practice
I don't know that it works well.  It would almost be better to have
some kind of cluster config file that specifies all the actual
dependencies (possibly including cross-host) and have it spit out all
the unit dependencies automatically.  That is a bit much to ask for
now, and probably a bit much for somebody who just wants their laptop
to launch kde after all their mounts are ready.

Specifying After vs Wants separately does make sense.  Dependency
doesn't have to imply sequential.

--
Rich

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Tom H]

On Fri, Oct 31, 2014 at 2:27 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Fri, Oct 31, 2014 at 1:34 PM, Tom H <tomh0665@gmail.com> wrote:


>> Is "After" really necessary as an option? I've never come across a
>> service that uses "After" without a "Requires" or a Wants" but I've
>> never taken the time to look.
>
> Hmm, I found After more common that Wants, but maybe I only look at
> units that have problems.  :)

LOL. Which supports the thesis that "After" might not be a useful
setting within a service unit. But it's just occured to me that target
units use "After" without "Requires" or "Wants", for example
network-online.target has "After=network.target".


> I think the intent is to handle optional dependencies, but in practice
> I don't know that it works well. It would almost be better to have
> some kind of cluster config file that specifies all the actual
> dependencies (possibly including cross-host) and have it spit out all
> the unit dependencies automatically. That is a bit much to ask for
> now, and probably a bit much for somebody who just wants their laptop
> to launch kde after all their mounts are ready.

Optional dependencies are handled by "Wants" like openrc's "use".

IIUC you're referring to a BSD-like rc daemon config file. WOuldn't
that have to be maintained by a sysadmin rather than by a package
maintainer?


> Specifying After vs Wants separately does make sense. Dependency
> doesn't have to imply sequential.

Do you have an example of a service that uses "After=" but doesn't
need a "Requires=" or a "Wants="? I'm either being unimaginative or
plain dumb, but I can't think of any. I wonder whether, if Lennart and
co removed "After=" from service units and turned "Requires=" into the
equivakent of the current "Requires=" and "After=" setup, someone
would raise a storm over the change because it would've broken
something.

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Rich Freeman]

On Fri, Oct 31, 2014 at 7:01 PM, Tom H <tomh0665@gmail.com> wrote:
>
> Do you have an example of a service that uses "After=" but doesn't
> need a "Requires=" or a "Wants="? I'm either being unimaginative or
> plain dumb, but I can't think of any.

Some examples I found:
smbd.service
sshd.service
mythbackend.service
ntpd.service

--
Rich

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Tom H]

On Fri, Oct 31, 2014 at 7:52 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Fri, Oct 31, 2014 at 7:01 PM, Tom H <tomh0665@gmail.com> wrote:
>>
>> Do you have an example of a service that uses "After=" but doesn't
>> need a "Requires=" or a "Wants="? I'm either being unimaginative or
>> plain dumb, but I can't think of any.
>
> sshd.service

Thanks. I've just looked at this one and it has "After=syslog.target
network.target auditd.service".

AIUI, "After=network.target" (and similarly "After=syslog.target") is
equivalent to having "Wants=network.service NetworkManager.service
<other_network_managers>" and "After=network.service
NetworkManager.service <other_network_managers>".

But "After=auditd.service" is clearly an example of an "After="
without a "Requires=" or a "Wants=". Thanks again.

PS: You'd expressed an desire for a dependency chain file in an
earlier email. "systemctl list-dependencies" shows you the current
dependency resolution. It's not what you were looking for but possibly
something of interest (it has modifiers like "--after" and
"--before").

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Rich Freeman]

On Sun, Nov 2, 2014 at 8:37 PM, Tom H <tomh0665@gmail.com> wrote:
>
> AIUI, "After=network.target" (and similarly "After=syslog.target") is
> equivalent to having "Wants=network.service NetworkManager.service
> <other_network_managers>" and "After=network.service
> NetworkManager.service <other_network_managers>".

Actually, as far as I understand things, if you don't enable something
that wants network.target, then you won't get it.  You need to enable
something like dhcpcd or networkd to get it.  It works this way so
that every package that needs the network doesn't try to run every
network manager you have installed as many are mutually exclusive.

Presumably if you don't run one of those services and start sshd, it
will just end up listening on localhost (I assume something sets up
the lo interface even if a network manager isn't running).

--
Rich

Re: [gentoo-user] Re: [~amd64] NFS server broken again :(

[Jc García]

2014-10-31 17:01 GMT-06:00 Tom H <tomh0665@gmail.com>:
> On Fri, Oct 31, 2014 at 2:27 PM, Rich Freeman <rich0@gentoo.org> wrote:
>> On Fri, Oct 31, 2014 at 1:34 PM, Tom H <tomh0665@gmail.com> wrote:
>
>
>>> Is "After" really necessary as an option? I've never come across a
>>> service that uses "After" without a "Requires" or a Wants" but I've
>>> never taken the time to look.
>>
>> Hmm, I found After more common that Wants, but maybe I only look at
>> units that have problems.  :)
>
> LOL. Which supports the thesis that "After" might not be a useful
> setting within a service unit. But it's just occured to me that target
> units use "After" without "Requires" or "Wants", for example
> network-online.target has "After=network.target".
>
I think the manuals are pretty clear about the working of these.
From the systemd.unit manual:
"""
....
Requires=
.... If a unit foo.service requires a unit bar.service as configured
with Requires= and no ordering is configured with After= or Before=,
then both units will be started simultaneously and without any delay
between them if foo.service is activated.

Before,After=
...
Note that this setting is independent of and orthogonal to the
requirement dependencies as configured by Requires=.
....
 If two units have no ordering dependencies between them, they are
shut down or started up simultaneously, and no ordering takes place.


"""



From sytemd.service manual
"""
 Unless DefaultDependencies= is set to false, service units will
implicitly have dependencies of type Requires= and After= on
       basic.target as well as dependencies of type Conflicts= and
Before= on shutdown.target. These ensure that normal service units
pull in
       basic system initialization, and are terminated cleanly prior
to system shutdown.

"""
I think it's about flexibility and the fact that systemd uses
parallelization at boot, when having these options makes sense

Re: [gentoo-user] [~amd64] NFS server broken again :(

[Tom H]

On Mon, Oct 27, 2014 at 3:38 PM, walt <w41ter@gmail.com> wrote:
>
> Last night when I powered off my machines NFS was working perfectly.  Today
> it's broken again for the nth time:
>
> #systemctl status nfs-server
> ● nfs-server.service - NFS server and services
>    Loaded: loaded (/usr/lib64/systemd/system/nfs-server.service; enabled)
>    Active: failed (Result: exit-code) since Mon 2014-10-27 11:50:38 PDT; 25min ago
>   Process: 896 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)
>   Process: 893 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
>   Process: 939 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE)
>   Process: 936 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
>  Main PID: 939 (code=exited, status=1/FAILURE)
>
> Oct 27 11:50:38 a6 rpc.nfsd[939]: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
> Oct 27 11:50:38 a6 rpc.nfsd[939]: rpc.nfsd: unable to set any sockets for nfsd
> Oct 27 11:50:38 a6 systemd[1]: nfs-server.service: main process exited, code=exited, status=1/FAILURE
> Oct 27 11:50:38 a6 systemd[1]: Failed to start NFS server and services.
> Oct 27 11:50:38 a6 systemd[1]: Unit nfs-server.service entered failed state.
>
> #rpc.nfsd -d
> rpc.nfsd: Checking netconfig for visible protocols.
> rpc.nfsd: Enabling inet udp.
> rpc.nfsd: Enabling inet tcp.
> rpc.nfsd: Enabling inet6 udp.
> rpc.nfsd: Enabling inet6 tcp.
> rpc.nfsd: knfsd is currently down
> rpc.nfsd: Writing version string to kernel: -2 +3 +4
> rpc.nfsd: Creating inet TCP socket.
> rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
> rpc.nfsd: Creating inet6 TCP socket.
> rpc.nfsd: writing fd to kernel failed: errno 97 (Address family not supported by protocol)
> rpc.nfsd: unable to set any sockets for nfsd

Is rpcbind.service running?

Does "rpcbind -p" output the usual programs/ports/...?

Are the nfs-api-filesystems mounted? ("/var/lib/nfs/rpc_pipefs" and
"/proc/fs/nfsd")