[gentoo-user] nfsv4 issues

[gentoo-user] nfsv4 issues

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 840 bytes --]

I'm trying to troubleshoot a newly setup nfs server, which, sometimes has a
30 second pause (tcpdump shows its server waiting).

# time touch /usr/portage/distfiles/testfile

real    0m30.088s
user    0m0.000s
sys     0m0.001s

I cant see anything in the nfs server debugging so i want to strace nfsd.
First i tell it to use single thread so i know i;m stracing the correct
thread, but;

# grep OPTS_RPC_NFSD nfs
#OPTS_RPC_NFSD="8 -N2 -V 3 -V 4 -V 4.1"
OPTS_RPC_NFSD="1 -N2 -V 3 -V 4 -V 4.1"
#  systemctl restart nfs-server
# pgrep -lf nfsd
23546 nfsd4_callbacks
23548 nfsd
23549 nfsd
23550 nfsd
23551 nfsd
23552 nfsd
23553 nfsd
23554 nfsd
23555 nfsd

So its not respecting the nproc setting. Any ideas? I also tried changing
EXPORTFS_TIMEOUT= since its currently set at 30. It didnt help, but perhaps
that's because its being ignored too.

[-- Attachment #2: Type: text/html, Size: 991 bytes --]

Re: [gentoo-user] nfsv4 issues

[Tom H]

On Tue, Jul 19, 2016 at 12:49 AM, Adam Carter <adamcarter3@gmail.com> wrote:
>
> I'm trying to troubleshoot a newly setup nfs server, which, sometimes has a
> 30 second pause (tcpdump shows its server waiting).
>
> # time touch /usr/portage/distfiles/testfile
>
> real    0m30.088s
> user    0m0.000s
> sys     0m0.001s
>
> I cant see anything in the nfs server debugging so i want to strace nfsd.
> First i tell it to use single thread so i know i;m stracing the correct
> thread, but;
>
> # grep OPTS_RPC_NFSD nfs
> #OPTS_RPC_NFSD="8 -N2 -V 3 -V 4 -V 4.1"
> OPTS_RPC_NFSD="1 -N2 -V 3 -V 4 -V 4.1"
> # systemctl restart nfs-server
> # pgrep -lf nfsd
> 23546 nfsd4_callbacks
> 23548 nfsd
> 23549 nfsd
> 23550 nfsd
> 23551 nfsd
> 23552 nfsd
> 23553 nfsd
> 23554 nfsd
> 23555 nfsd
>
> So its not respecting the nproc setting. Any ideas? I also tried changing
> EXPORTFS_TIMEOUT= since its currently set at 30. It didnt help, but perhaps
> that's because its being ignored too.

Are the nfsd versions that you're setting being respected? You can
check with "rpcinfo -s" or "cat /proc/fs/nfsd/versions".

You can change the number of threads on the fly with "echo 1 >
/proc/fs/nfsd/threads".

I don't use systemd on Gentoo but for the nfs-utils upstream-shipped
systemd units that I think that Gentoo's using, you have to re-run
nfs-config.service - or run the script that it calls - in order to
update the "/run/sysconfig/nfs-utils" environment file that's sourced
by the nfs-server.service unit.

Does "/var/lib/nfs/v4recovery/" exist?

Does adding the client to "/etc/hosts" - or to your reverse dns zone -
eliminate the delay?

Re: [gentoo-user] nfsv4 issues

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 1325 bytes --]

Are the nfsd versions that you're setting being respected? You can

> check with "rpcinfo -s" or "cat /proc/fs/nfsd/versions".
>

Yep;
 # cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 +4.2


> You can change the number of threads on the fly with "echo 1 >
> /proc/fs/nfsd/threads".
>

That works too, but then;
# ps -ef | grep nfsd
root      1454  1426  0 12:47 pts/0    00:00:00 grep --colour=auto nfsd
root     23546     2  0 Jul19 ?        00:00:00 [nfsd4_callbacks]
root     23548     2  0 Jul19 ?        00:00:00 [nfsd]
# strace -p 23548
strace: attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted


> I don't use systemd on Gentoo but for the nfs-utils upstream-shipped
> systemd units that I think that Gentoo's using, you have to re-run
> nfs-config.service - or run the script that it calls - in order to
> update the "/run/sysconfig/nfs-utils" environment file that's sourced
> by the nfs-server.service unit.
>

In /usr/lib/systemd/system/nfs-server.service
[Service]
EnvironmentFile=/etc/conf.d/nfs

Does "/var/lib/nfs/v4recovery/" exist?
>
>
No
 # ls /var/lib/nfs/
etab  export-lock  rmtab  rpc_pipefs  sm  sm.bak  state  xtab

Does adding the client to "/etc/hosts" - or to your reverse dns zone -
> eliminate the delay?
>
> DNS is setup and both client and server can forward and reverse lookup
each other.

[-- Attachment #2: Type: text/html, Size: 2451 bytes --]

Re: [gentoo-user] nfsv4 issues

[Tom H]

On Wed, Jul 20, 2016 at 10:51 PM, Adam Carter <adamcarter3@gmail.com> wrote:


>> I don't use systemd on Gentoo but for the nfs-utils upstream-shipped
>> systemd units that I think that Gentoo's using, you have to re-run
>> nfs-config.service - or run the script that it calls - in order to
>> update the "/run/sysconfig/nfs-utils" environment file that's sourced
>> by the nfs-server.service unit.
>
> In /usr/lib/systemd/system/nfs-server.service
> [Service]
> EnvironmentFile=/etc/conf.d/nfs

Sorry. Looking at the ebuild, there's:

<begin>
rm "${D}$(systemd_get_unitdir)"/nfs-config.service || die
sed -i -r \
-e "/^EnvironmentFile=/s:=.*:=${EPREFIX}/etc/conf.d/nfs:" \
-e '/^(After|Wants)=nfs-config.service$/d' \
-e 's:/usr/sbin/rpc.statd:/sbin/rpc.statd:' \
"${D}$(systemd_get_unitdir)"/* || die
</end>

so the upstream "nfs-config.service" waltz is avoided.

But that means that the variables in "/etc/conf.d/nfs" aren't renamed.
So the openrc nfs script uses "${OPTS_RPC_NFSD}", which is defined,
and the systemd service uses "$RPCNFSDARGS", which isn't.


>> Does "/var/lib/nfs/v4recovery/" exist?
>
> No
> # ls /var/lib/nfs/
> etab export-lock rmtab rpc_pipefs sm sm.bak state xtab

IIRC, it's needed to avoid this delay. I thought that I'd saved a url
about this but I can't find it.

Do you have a syslog message about "stable storage"? "man nfsdcltrack".

The openrc script has

<begin>
mkdir_nfsdirs() {
local d
for d in v4recovery v4root ; do
d="/var/lib/nfs/${d}"
[ ! -d "${d}" ] && mkdir -p "${d}"
done
}
</end>

but systemd doesn't have anything equivalent. On RHEL and Ubuntu,
"/var/lib/nfs/v4recovery/" is created at installation time. Perhaps
the Gentoo ebuild should do the same or should ship a
"/usr/lib/tmpfiles.d/var-lib-nfs.conf" to create it at boot if it
doesn't exist.

Re: [gentoo-user] nfsv4 issues

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 2638 bytes --]

> >> I don't use systemd on Gentoo but for the nfs-utils upstream-shipped
> >> systemd units that I think that Gentoo's using, you have to re-run
> >> nfs-config.service - or run the script that it calls - in order to
> >> update the "/run/sysconfig/nfs-utils" environment file that's sourced
> >> by the nfs-server.service unit.
> >
> > In /usr/lib/systemd/system/nfs-server.service
> > [Service]
> > EnvironmentFile=/etc/conf.d/nfs
>
> Sorry. Looking at the ebuild, there's:
>
> <begin>
> rm "${D}$(systemd_get_unitdir)"/nfs-config.service || die
> sed -i -r \
> -e "/^EnvironmentFile=/s:=.*:=${EPREFIX}/etc/conf.d/nfs:" \
> -e '/^(After|Wants)=nfs-config.service$/d' \
> -e 's:/usr/sbin/rpc.statd:/sbin/rpc.statd:' \
> "${D}$(systemd_get_unitdir)"/* || die
> </end>
>
> so the upstream "nfs-config.service" waltz is avoided.
>
> But that means that the variables in "/etc/conf.d/nfs" aren't renamed.
> So the openrc nfs script uses "${OPTS_RPC_NFSD}", which is defined,
> and the systemd service uses "$RPCNFSDARGS", which isn't.
>

I've added $RPCNFSDARGS to /etc/conf.d/nfs, restarted, and the nproc
setting works.

>
> >> Does "/var/lib/nfs/v4recovery/" exist?
> >
> > No
> > # ls /var/lib/nfs/
> > etab export-lock rmtab rpc_pipefs sm sm.bak state xtab
>
> IIRC, it's needed to avoid this delay. I thought that I'd saved a url
> about this but I can't find it.
>
> Do you have a syslog message about "stable storage"? "man nfsdcltrack".
>

There's no message about stable storage, but there's this;
kernel: [578030.628415] NFSD: the nfsdcld client tracking upcall will be
removed in 3.10. Please transition to using nfsdcltrack.

# which nfsdcltrack
which: no nfsdcltrack in
(/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/5.4.0:/usr/lib64/subversion/bin:/opt/vmware/bin)
# qlist nfs | grep nfsdcltrack
#


> The openrc script has
>
> <begin>
> mkdir_nfsdirs() {
> local d
> for d in v4recovery v4root ; do
> d="/var/lib/nfs/${d}"
> [ ! -d "${d}" ] && mkdir -p "${d}"
> done
> }
> </end>
>
> but systemd doesn't have anything equivalent. On RHEL and Ubuntu,
> "/var/lib/nfs/v4recovery/" is created at installation time. Perhaps
> the Gentoo ebuild should do the same or should ship a
> "/usr/lib/tmpfiles.d/var-lib-nfs.conf" to create it at boot if it
> doesn't exist.
>
> I've added the directory, and after restarting syslog now has new entries;
kernel: [912267.948883] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4
state recovery directory
kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery
directory

I will test shortly and report back - thanks!

[-- Attachment #2: Type: text/html, Size: 3888 bytes --]

Re: [gentoo-user] nfsv4 issues

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

I've added the directory, and after restarting syslog now has new entries;

> kernel: [912267.948883] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4
> state recovery directory
> kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery
> directory
>
> I will test shortly and report back - thanks!
>

Confirmed - this fixes the 30 second delay. Should i log a bug for these
issues?

[-- Attachment #2: Type: text/html, Size: 719 bytes --]

Re: [gentoo-user] nfsv4 issues

[Tom H]

On Sun, Jul 24, 2016 at 3:37 AM, Adam Carter <adamcarter3@gmail.com> wrote:


> I've added the directory, and after restarting syslog now has new entries;
>>
>> kernel: [912267.948883] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4
>> state recovery directory
>> kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery
>> directory
>>
>> I will test shortly and report back - thanks!
>
> Confirmed - this fixes the 30 second delay.

Good.


> Should i log a bug for these issues?

If I were you, I'd definitely file a bug reports against nfs-utils for:

1) the creation of "/var/lib/nfs/v4recovery/" when systemd is pid 1;

2) the systemd unit compatible envvars.

Re: [gentoo-user] nfsv4 issues

[Tom H]

On Fri, Jul 22, 2016 at 8:22 PM, Adam Carter <adamcarter3@gmail.com> wrote:


>>>> Does "/var/lib/nfs/v4recovery/" exist?
>>>
>>> No
>>> # ls /var/lib/nfs/
>>> etab export-lock rmtab rpc_pipefs sm sm.bak state xtab
>>
>> IIRC, it's needed to avoid this delay. I thought that I'd saved a url
>> about this but I can't find it.
>>
>> Do you have a syslog message about "stable storage"? "man nfsdcltrack".
>
> There's no message about stable storage, but there's this;
> kernel: [578030.628415] NFSD: the nfsdcld client tracking upcall will be
> removed in 3.10. Please transition to using nfsdcltrack.

It's from

https://patchwork.kernel.org/patch/1730241/


> # which nfsdcltrack
> which: no nfsdcltrack in
> (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/5.4.0:/usr/lib64/subversion/bin:/opt/vmware/bin)
> # qlist nfs | grep nfsdcltrack
> #

It depends on the nfs-utils USE settings:

# qlist -U nfs-utils
net-fs/nfs-utils (libmount nfsdcld nfsidmap nfsv4 nfsv41)

# qfile $(which nfsdcltrack)
net-fs/nfs-utils (/sbin/nfsdcltrack)


>> The openrc script has
>>
>> <begin>
>> mkdir_nfsdirs() {
>> local d
>> for d in v4recovery v4root ; do
>> d="/var/lib/nfs/${d}"
>> [ ! -d "${d}" ] && mkdir -p "${d}"
>> done
>> }
>> </end>
>>
>> but systemd doesn't have anything equivalent. On RHEL and Ubuntu,
>> "/var/lib/nfs/v4recovery/" is created at installation time. Perhaps
>> the Gentoo ebuild should do the same or should ship a
>> "/usr/lib/tmpfiles.d/var-lib-nfs.conf" to create it at boot if it
>> doesn't exist.
>
> I've added the directory, and after restarting syslog now has new entries;
> kernel: [912267.948883] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4
> state recovery directory
> kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery
> directory
>
> I will test shortly and report back - thanks!

Good luck. You're welcome.