[gentoo-user] NFS tutorial for the brain dead sysadmin?

[gentoo-user] NFS tutorial for the brain dead sysadmin?

[walt]

In this case, the brain dead sysadmin would be moi :)

For years I've been using NFS to share /usr/portage with all of the
gentoo machines on my LAN.

Problem:  occasionally it stops working for no apparent reason.

Example:  two days ago I updated two ~amd64 gentoo machines, both of
which have been mounting /usr/portage as NFS3 shares for at least a
year with no problems.

One machine worked normally after the update, the other was unable to
mount /usr/portage because rpc.statd wouldn't start correctly.

After two frustrating days I discovered that I had never enabled the
rpcbind.service on the "broken" machine.  So I enabled rpcbind, which
fixed the breakage.

So, why did the "broken" machine work normally for more than a year
without rpcbind until two days ago?  (I suppose because nfs-utils was
updated to 1.3.0 ?)

The real problem here is that I have no idea how NFS works, and each
new version is more complicated because the devs are solving problems
that I don't understand or even know about.

So, please, what's the best way to learn and understand NFS?

Thanks for any clues.

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[Alan McKinnon]

On 26/07/2014 04:47, walt wrote:
> In this case, the brain dead sysadmin would be moi :)
> 
> For years I've been using NFS to share /usr/portage with all of the
> gentoo machines on my LAN.
> 
> Problem:  occasionally it stops working for no apparent reason.
> 
> Example:  two days ago I updated two ~amd64 gentoo machines, both of
> which have been mounting /usr/portage as NFS3 shares for at least a
> year with no problems.
> 
> One machine worked normally after the update, the other was unable to
> mount /usr/portage because rpc.statd wouldn't start correctly.
> 
> After two frustrating days I discovered that I had never enabled the
> rpcbind.service on the "broken" machine.  So I enabled rpcbind, which
> fixed the breakage.
> 
> So, why did the "broken" machine work normally for more than a year
> without rpcbind until two days ago?  (I suppose because nfs-utils was
> updated to 1.3.0 ?)
> 
> The real problem here is that I have no idea how NFS works, and each
> new version is more complicated because the devs are solving problems
> that I don't understand or even know about.
> 
> So, please, what's the best way to learn and understand NFS?


I think you are asking for the impossible :-)

NFS is not easy to set up, and even harder to describe. It is easy to
*use* once it's set up correctly - you just mount something and the only
difference to a local mount is you add an IP address.

NFS uses RPC to do some heavy lifting - I don't know how familiar you
are with this, so here's the quick version:

When you mount something locally, and need to use the mounted
filesystem, kernel calls are used to get at the data. This works easily
as the source disk is local and the kernel can get to it. With NFS, the
source disk is remote and it's the remote kernel that must do the
accessing. RPC is a way to safely ask a remote kernel to do something
and get a result that behaves identical to a local kernel call.
Obviously, this is rather hard to implement correctly.

The original RPC was written by Sun and other newer implementations
exist, like libtirpc - to support useful features like not being stuck
with only UDP. That's what the "ti" means - Transport Independant.

RPC has been in a state of flux for some time and I too have run into
init-script oddities as things change.

In my case, I have nfs-utils-1.3.0, and rc-update configuredd to start
rpc.statd. This works because

depend() {
        ...
        need portmap
        ...
}

and in the init.d file for rpcbind:

depend() {
        ...
        provide portmap
}

So rpcbind starts at boot time and all my nfs mounts JustWork

Looks to me like your problem is actually with rpc and more specifically
with what things are currently named today (which could be different to
yesterday). Unfortunately I don't know of a place where this is all
nicely described in a sane manner except inside the init files themselves.



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[Tom H]

On Sat, Jul 26, 2014 at 9:51 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>
> NFS uses RPC to do some heavy lifting - I don't know how familiar you
> are with this, so here's the quick version:
>
> When you mount something locally, and need to use the mounted
> filesystem, kernel calls are used to get at the data. This works easily
> as the source disk is local and the kernel can get to it. With NFS, the
> source disk is remote and it's the remote kernel that must do the
> accessing. RPC is a way to safely ask a remote kernel to do something
> and get a result that behaves identical to a local kernel call.
> Obviously, this is rather hard to implement correctly.
>
> The original RPC was written by Sun and other newer implementations
> exist, like libtirpc - to support useful features like not being stuck
> with only UDP. That's what the "ti" means - Transport Independant.
>
> RPC has been in a state of flux for some time and I too have run into
> init-script oddities as things change.
>
> In my case, I have nfs-utils-1.3.0, and rc-update configuredd to start
> rpc.statd. This works because
>
> depend() {
>     ...
>     need portmap
>     ...
> }
>
> and in the init.d file for rpcbind:
>
> depend() {
>     ...
>     provide portmap
> }
>
> So rpcbind starts at boot time and all my nfs mounts JustWork

To confirm the above, for nfs-utils-1.2.9-r3.

If I start nfs manually, all the associated daemons start too even
though I haven't added them to "default" (although idmapd is started
because of "/etc/conf.d/nfs"):

# ls -1 /etc/init.d/rpc*
/etc/init.d/rpc.idmapd
/etc/init.d/rpc.pipefs
/etc/init.d/rpc.statd
/etc/init.d/rpcbind

# rc-update | grep rpc

# rc-service nfs start
 * Starting rpcbind ...                  [ ok ]
 * Starting NFS statd ...                [ ok ]
 * Setting up RPC pipefs ...             [ ok ]
 * Starting idmapd ...                   [ ok ]
 * Mounting nfsd filesystem in /proc ... [ ok ]
 * Exporting NFS directories ...         [ ok ]
 * Starting NFS mountd ...               [ ok ]
 * Starting NFS daemon ...               [ ok ]
 * Starting NFS smnotify ...             [ ok ]

#

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[Stefan G. Weichinger]

Am 26.07.2014 04:47, schrieb walt:

> So, why did the "broken" machine work normally for more than a year
> without rpcbind until two days ago?  (I suppose because nfs-utils was
> updated to 1.3.0 ?)
> 
> The real problem here is that I have no idea how NFS works, and each
> new version is more complicated because the devs are solving problems
> that I don't understand or even know about.

I double your search for understanding ... my various efforts to set up
NFSv4 for sharing stuff in my LAN also lead to unstable behavior and
frustration.

Only last week I re-attacked this topic as I start using puppet here to
manage my systems ... and one part of this might be sharing /usr/portage
via NFSv4. One client host mounts it without a problem, the thinkpads
don't do so ... just another example ;-)

Additional in my context: using systemd ... so there are other
(different?) dependencies at work and services started.

I'd be happy to get that working in a reliable way. I don't remember
unstable behavior with NFS (v2 back then?) when we used it at a company
I worked for in the 90s.

Stefan

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[J. Roeleveld]

On 27 July 2014 18:25:24 CEST, "Stefan G. Weichinger" <lists@xunil.at> wrote:
>Am 26.07.2014 04:47, schrieb walt:
>
>> So, why did the "broken" machine work normally for more than a year
>> without rpcbind until two days ago?  (I suppose because nfs-utils was
>> updated to 1.3.0 ?)
>> 
>> The real problem here is that I have no idea how NFS works, and each
>> new version is more complicated because the devs are solving problems
>> that I don't understand or even know about.
>
>I double your search for understanding ... my various efforts to set up
>NFSv4 for sharing stuff in my LAN also lead to unstable behavior and
>frustration.
>
>Only last week I re-attacked this topic as I start using puppet here to
>manage my systems ... and one part of this might be sharing
>/usr/portage
>via NFSv4. One client host mounts it without a problem, the thinkpads
>don't do so ... just another example ;-)
>
>Additional in my context: using systemd ... so there are other
>(different?) dependencies at work and services started.
>
>I'd be happy to get that working in a reliable way. I don't remember
>unstable behavior with NFS (v2 back then?) when we used it at a company
>I worked for in the 90s.
>
>Stefan

I use NFS for filesharing between all wired systems at home.
Samba is only used for MS Windows and laptops.

Few things I always make sure are valid:
- One partition per NFS share
- No NFS share is mounted below another one
- I set the version to 3 on the clients
- I use LDAP for the user accounts to ensure the UIDs and GIDs are consistent.

NFS4 requires all the exports to be under a single foldertree.

I haven't had any issues in the past 7+ years with this and in the past 5+ years I had portage, distfiles and packages shared.
/etc/portage is symlinked to a NFS share as well, allowing me to create binary packages on a single host (inside a chroot) which are then used to update the different machines.

If anyone wants a more detailed description of my setup. Let me know and I will try to write something up.

Kind regards

Joost

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[Kerin Millar]

On 27/07/2014 17:55, J. Roeleveld wrote:
> On 27 July 2014 18:25:24 CEST, "Stefan G. Weichinger" <lists@xunil.at> wrote:
>> Am 26.07.2014 04:47, schrieb walt:
>>
>>> So, why did the "broken" machine work normally for more than a year
>>> without rpcbind until two days ago?  (I suppose because nfs-utils was
>>> updated to 1.3.0 ?)
>>>
>>> The real problem here is that I have no idea how NFS works, and each
>>> new version is more complicated because the devs are solving problems
>>> that I don't understand or even know about.
>>
>> I double your search for understanding ... my various efforts to set up
>> NFSv4 for sharing stuff in my LAN also lead to unstable behavior and
>> frustration.
>>
>> Only last week I re-attacked this topic as I start using puppet here to
>> manage my systems ... and one part of this might be sharing
>> /usr/portage
>> via NFSv4. One client host mounts it without a problem, the thinkpads
>> don't do so ... just another example ;-)
>>
>> Additional in my context: using systemd ... so there are other
>> (different?) dependencies at work and services started.
>>
>> I'd be happy to get that working in a reliable way. I don't remember
>> unstable behavior with NFS (v2 back then?) when we used it at a company
>> I worked for in the 90s.
>>
>> Stefan
>
> I use NFS for filesharing between all wired systems at home.
> Samba is only used for MS Windows and laptops.
>
> Few things I always make sure are valid:
> - One partition per NFS share
> - No NFS share is mounted below another one
> - I set the version to 3 on the clients
> - I use LDAP for the user accounts to ensure the UIDs and GIDs are consistent.

These are generally good recommendations. I'd just like to make a few 
observations.

The problems associated with not observing the first constraint (one 
filesystem per export) can be alleviated by setting an explicit fsid. 
Doing so can also help to avoid stale handles on the client side if the 
backing filesystem changes - something that is very useful in a 
production environment. Therefore, I tend to start at 1 and increment 
with each newly added export. For example:-

   /export/foo      *(async,no_subtree_check,fsid=1)
   /export/foo/bar  *(async,no_subtree_check,fsid=2)
   /export/baz      *(async,no_subtree_check,fsid=3)

If using NFSv3, I'd recommend using "nolock" as a mount option unless 
there is a genuine requirement for locks to be co-ordinated. Such locks 
are only advisory and are of questionable value. Using nolock simplifies 
the requirements on both server and client side, and is beneficial for 
performance.

NFSv3/UDP seems to be limited to a maximum read/write block size of 
32768 in Linux, which will be negotiated by default. Using TCP, the 
upper bound will be the value of /proc/fs/nfsd/max_block_size on the 
server. Its value may be set to 1048576 at the most. NFSv3/TCP is 
problematic so I would recommend NFSv4 if TCP is desired as a transport 
protocol.

NFSv4 provides a useful uid/gid mapping feature that is easier to set up 
and maintain than nss_ldap.

>
> NFS4 requires all the exports to be under a single foldertree.

This is a myth: 
http://linuxcostablanca.blogspot.co.uk/2012/02/nfsv4-myths-and-legends.html. 
Exports can be defined and consumed in the same manner as with NFSv3.

>
> I haven't had any issues in the past 7+ years with this and in the past 5+ years I had portage, distfiles and packages shared.
> /etc/portage is symlinked to a NFS share as well, allowing me to create binary packages on a single host (inside a chroot) which are then used to update the different machines.
>
> If anyone wants a more detailed description of my setup. Let me know and I will try to write something up.
>
> Kind regards
>
> Joost
>

--Kerin

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[J. Roeleveld]

On Sunday, July 27, 2014 08:44:02 PM Kerin Millar wrote:
> On 27/07/2014 17:55, J. Roeleveld wrote:
> > On 27 July 2014 18:25:24 CEST, "Stefan G. Weichinger" <lists@xunil.at> 
wrote:
> >> Am 26.07.2014 04:47, schrieb walt:
> >>> So, why did the "broken" machine work normally for more than a year
> >>> without rpcbind until two days ago?  (I suppose because nfs-utils was
> >>> updated to 1.3.0 ?)
> >>> 
> >>> The real problem here is that I have no idea how NFS works, and each
> >>> new version is more complicated because the devs are solving problems
> >>> that I don't understand or even know about.
> >> 
> >> I double your search for understanding ... my various efforts to set up
> >> NFSv4 for sharing stuff in my LAN also lead to unstable behavior and
> >> frustration.
> >> 
> >> Only last week I re-attacked this topic as I start using puppet here to
> >> manage my systems ... and one part of this might be sharing
> >> /usr/portage
> >> via NFSv4. One client host mounts it without a problem, the thinkpads
> >> don't do so ... just another example ;-)
> >> 
> >> Additional in my context: using systemd ... so there are other
> >> (different?) dependencies at work and services started.
> >> 
> >> I'd be happy to get that working in a reliable way. I don't remember
> >> unstable behavior with NFS (v2 back then?) when we used it at a company
> >> I worked for in the 90s.
> >> 
> >> Stefan
> > 
> > I use NFS for filesharing between all wired systems at home.
> > Samba is only used for MS Windows and laptops.
> > 
> > Few things I always make sure are valid:
> > - One partition per NFS share
> > - No NFS share is mounted below another one
> > - I set the version to 3 on the clients
> > - I use LDAP for the user accounts to ensure the UIDs and GIDs are
> > consistent.
> These are generally good recommendations. I'd just like to make a few
> observations.
> 
> The problems associated with not observing the first constraint (one
> filesystem per export) can be alleviated by setting an explicit fsid.
> Doing so can also help to avoid stale handles on the client side if the
> backing filesystem changes - something that is very useful in a
> production environment. Therefore, I tend to start at 1 and increment
> with each newly added export. For example:-
> 
>    /export/foo      *(async,no_subtree_check,fsid=1)
>    /export/foo/bar  *(async,no_subtree_check,fsid=2)
>    /export/baz      *(async,no_subtree_check,fsid=3)
> 
> If using NFSv3, I'd recommend using "nolock" as a mount option unless
> there is a genuine requirement for locks to be co-ordinated. Such locks
> are only advisory and are of questionable value. Using nolock simplifies
> the requirements on both server and client side, and is beneficial for
> performance.
> 
> NFSv3/UDP seems to be limited to a maximum read/write block size of
> 32768 in Linux, which will be negotiated by default. Using TCP, the
> upper bound will be the value of /proc/fs/nfsd/max_block_size on the
> server. Its value may be set to 1048576 at the most. NFSv3/TCP is
> problematic so I would recommend NFSv4 if TCP is desired as a transport
> protocol.
> 
> NFSv4 provides a useful uid/gid mapping feature that is easier to set up
> and maintain than nss_ldap.
> 
> > NFS4 requires all the exports to be under a single foldertree.
> 
> This is a myth:
> http://linuxcostablanca.blogspot.co.uk/2012/02/nfsv4-myths-and-legends.html.
> Exports can be defined and consumed in the same manner as with NFSv3.

When I originally tried NFSv4, it refused to work unless they were all under 
the same directory.
As I dislike that, I decided against using it.

That was a long time ago, will revisit that part again later.

Interesting link, I wonder how difficult it will be to combine that with Samba 
4 and use the Samba AD structure for NFSv4 with either ZFS or BTRFS 
underneath.

--
Joost

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[behrouz khosravi]

[-- Attachment #1: Type: text/plain, Size: 4345 bytes --]

Hello every body.
I was wondering that is it possible to make portage to sync a only a subset
of portage tree. For example I have not installed Gnome and I dont want to
sysc command download ebuilds related to this branch.
thanks


On Mon, Jul 28, 2014 at 6:28 PM, J. Roeleveld <joost@antarean.org> wrote:

> On Sunday, July 27, 2014 08:44:02 PM Kerin Millar wrote:
> > On 27/07/2014 17:55, J. Roeleveld wrote:
> > > On 27 July 2014 18:25:24 CEST, "Stefan G. Weichinger" <lists@xunil.at>
> wrote:
> > >> Am 26.07.2014 04:47, schrieb walt:
> > >>> So, why did the "broken" machine work normally for more than a year
> > >>> without rpcbind until two days ago?  (I suppose because nfs-utils was
> > >>> updated to 1.3.0 ?)
> > >>>
> > >>> The real problem here is that I have no idea how NFS works, and each
> > >>> new version is more complicated because the devs are solving problems
> > >>> that I don't understand or even know about.
> > >>
> > >> I double your search for understanding ... my various efforts to set
> up
> > >> NFSv4 for sharing stuff in my LAN also lead to unstable behavior and
> > >> frustration.
> > >>
> > >> Only last week I re-attacked this topic as I start using puppet here
> to
> > >> manage my systems ... and one part of this might be sharing
> > >> /usr/portage
> > >> via NFSv4. One client host mounts it without a problem, the thinkpads
> > >> don't do so ... just another example ;-)
> > >>
> > >> Additional in my context: using systemd ... so there are other
> > >> (different?) dependencies at work and services started.
> > >>
> > >> I'd be happy to get that working in a reliable way. I don't remember
> > >> unstable behavior with NFS (v2 back then?) when we used it at a
> company
> > >> I worked for in the 90s.
> > >>
> > >> Stefan
> > >
> > > I use NFS for filesharing between all wired systems at home.
> > > Samba is only used for MS Windows and laptops.
> > >
> > > Few things I always make sure are valid:
> > > - One partition per NFS share
> > > - No NFS share is mounted below another one
> > > - I set the version to 3 on the clients
> > > - I use LDAP for the user accounts to ensure the UIDs and GIDs are
> > > consistent.
> > These are generally good recommendations. I'd just like to make a few
> > observations.
> >
> > The problems associated with not observing the first constraint (one
> > filesystem per export) can be alleviated by setting an explicit fsid.
> > Doing so can also help to avoid stale handles on the client side if the
> > backing filesystem changes - something that is very useful in a
> > production environment. Therefore, I tend to start at 1 and increment
> > with each newly added export. For example:-
> >
> >    /export/foo      *(async,no_subtree_check,fsid=1)
> >    /export/foo/bar  *(async,no_subtree_check,fsid=2)
> >    /export/baz      *(async,no_subtree_check,fsid=3)
> >
> > If using NFSv3, I'd recommend using "nolock" as a mount option unless
> > there is a genuine requirement for locks to be co-ordinated. Such locks
> > are only advisory and are of questionable value. Using nolock simplifies
> > the requirements on both server and client side, and is beneficial for
> > performance.
> >
> > NFSv3/UDP seems to be limited to a maximum read/write block size of
> > 32768 in Linux, which will be negotiated by default. Using TCP, the
> > upper bound will be the value of /proc/fs/nfsd/max_block_size on the
> > server. Its value may be set to 1048576 at the most. NFSv3/TCP is
> > problematic so I would recommend NFSv4 if TCP is desired as a transport
> > protocol.
> >
> > NFSv4 provides a useful uid/gid mapping feature that is easier to set up
> > and maintain than nss_ldap.
> >
> > > NFS4 requires all the exports to be under a single foldertree.
> >
> > This is a myth:
> >
> http://linuxcostablanca.blogspot.co.uk/2012/02/nfsv4-myths-and-legends.html
> .
> > Exports can be defined and consumed in the same manner as with NFSv3.
>
> When I originally tried NFSv4, it refused to work unless they were all
> under
> the same directory.
> As I dislike that, I decided against using it.
>
> That was a long time ago, will revisit that part again later.
>
> Interesting link, I wonder how difficult it will be to combine that with
> Samba
> 4 and use the Samba AD structure for NFSv4 with either ZFS or BTRFS
> underneath.
>
> --
> Joost
>
>

[-- Attachment #2: Type: text/html, Size: 5580 bytes --]

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

On Mon, 28 Jul 2014 19:59:16 +0430, behrouz khosravi wrote:

> I was wondering that is it possible to make portage to sync a only a
> subset of portage tree. For example I have not installed Gnome and I
> dont want to sysc command download ebuilds related to this branch.

Please do not top-post
Please do not hijack threads.

If you have a new question to ask, start a new thread, don't use a thread
dedicated to a different question.

The short answer to your question is "no" - unless you want to start
messing with RSYNC_OPTS in make.conf to add exclude directives, but that
could break dependency resolution.


-- 
Neil Bothwick

This is a test of the emergency tagline stealing system.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[behrouz khosravi]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

oh my bad!
Believe me, I did an honest mistake! and I am very sorry for that.
Thanks for you help and again, may apologies.


On Mon, Jul 28, 2014 at 8:27 PM, Neil Bothwick <neil@digimed.co.uk> wrote:

> On Mon, 28 Jul 2014 19:59:16 +0430, behrouz khosravi wrote:
>
> > I was wondering that is it possible to make portage to sync a only a
> > subset of portage tree. For example I have not installed Gnome and I
> > dont want to sysc command download ebuilds related to this branch.
>
> Please do not top-post
> Please do not hijack threads.
>
> If you have a new question to ask, start a new thread, don't use a thread
> dedicated to a different question.
>
> The short answer to your question is "no" - unless you want to start
> messing with RSYNC_OPTS in make.conf to add exclude directives, but that
> could break dependency resolution.
>
>
> --
> Neil Bothwick
>
> This is a test of the emergency tagline stealing system.
>

[-- Attachment #2: Type: text/html, Size: 1398 bytes --]

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[Peter Humphrey]

On Tuesday 29 July 2014 12:19:14 behrouz khosravi wrote:
> oh my bad!
> Believe me, I did an honest mistake! and I am very sorry for that.
> Thanks for you help and again, may apologies.
> 
> On Mon, Jul 28, 2014 at 8:27 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> > On Mon, 28 Jul 2014 19:59:16 +0430, behrouz khosravi wrote:
> > > I was wondering that is it possible to make portage to sync a only a
> > > subset of portage tree. For example I have not installed Gnome and I
> > > dont want to sysc command download ebuilds related to this branch.
> > 
> > Please do not top-post
> > Please do not hijack threads.
> > 
> > If you have a new question to ask, start a new thread, don't use a thread
> > dedicated to a different question.
> > 
> > The short answer to your question is "no" - unless you want to start
> > messing with RSYNC_OPTS in make.conf to add exclude directives, but that
> > could break dependency resolution.

And besides, a few gnome packages are needed by other parts of the system. My 
KDE box, for example, has gnome-base/libglade and gnome-base/librsvg:


So you shouldn't want to do it anyway.

-- 
Regards
Peter

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[behrouz khosravi]

[-- Attachment #1: Type: text/plain, Size: 41 bytes --]

oh, I guess I have to let it be!
Thanks.

[-- Attachment #2: Type: text/html, Size: 72 bytes --]

Re: [gentoo-user] NFS tutorial for the brain dead sysadmin?

[Stefan G. Weichinger]

Am 27.07.2014 18:25, schrieb Stefan G. Weichinger:

> Only last week I re-attacked this topic as I start using puppet here to
> manage my systems ... and one part of this might be sharing /usr/portage
> via NFSv4. One client host mounts it without a problem, the thinkpads
> don't do so ... just another example ;-)

As so often ... my fault: thinkpads did have NFSv4 in the kernel, but no
nfs-utils installed ... ;-)

sorry, S