[gentoo-user] /home doesn't umount on shutdown

[gentoo-user] /home doesn't umount on shutdown

[Alan McKinnon]

On the rare occasion when I reboot or shut this laptop down, it
continually and consistently gets stuck on one of the final steps, to
umount /home

The process never proceeds beyond that point (as /home is always
fsck'ed on next startup). I can't see any logs as syslog has already
been shut down at this point, and it happens whether I shutdown as root
from the console or by using the KDE widget.

/home here is on LVM

I could probably debug this easily enough if I could determine how the
shutdown sequence is ordered, or get a verbose output. But sadly, my fu
for such stuff has run out.

Anyone got pointers on where to start poking around?

[I'm not looking for solutions, I'm unlikely to get those right off the
bat, just looking for pointers atm]



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] /home doesn't umount on shutdown

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 892 bytes --]

On Tue, 15 Jan 2013 11:57:21 +0200, Alan McKinnon wrote:

> On the rare occasion when I reboot or shut this laptop down, it
> continually and consistently gets stuck on one of the final steps, to
> umount /home

If you logout as your user(s) so only root is logged in, does lsof show
any hits for /home?

I had a similar problem with my MythTV backend failing to unmount /var.
It turned out that mythbackend was failing to shutdown but openrc carried
on trying to shutdown. If I make sure mythbackend really is stopped, the
reboot proceeds normally, which is much better since I had to go into the
loft to reboot the box manually,


-- 
Neil Bothwick

When told the reason for Daylight Saving time the old Indian said...
"Only a white man would believe that you could cut a foot off the top of a
blanket And sew it to the bottom of a blanket and have a longer blanket."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] /home doesn't umount on shutdown

[Alan McKinnon]

On Tue, 15 Jan 2013 11:09:56 +0000
Neil Bothwick <neil@digimed.co.uk> wrote:

> On Tue, 15 Jan 2013 11:57:21 +0200, Alan McKinnon wrote:
> 
> > On the rare occasion when I reboot or shut this laptop down, it
> > continually and consistently gets stuck on one of the final steps,
> > to umount /home
> 
> If you logout as your user(s) so only root is logged in, does lsof
> show any hits for /home?

Only 1 hit - a background ssh process that sets up a bunch of tunnels
and port forwards so I can get into the corporate network for anywhere.

But I was not able to make the problem re-appear in short reboot
cycles. So whatever is hanging the box is something that starts up in
the course of work, it doesn't appear to be there directly after a KDE
login.

I might have to look at my work flow closely to find all those unusual
things I do in the course of work. I did manage to see the full
shutdown output on the screen with these tests though - it
umounts /home then umounts /boot, but the second umount message is
never displayed. Seeing as it's just a regular sdb1 partition using
ext2 I don;t really think it's getting stuck as the next step starts
up. It's more likely to have something to do with /home


> 
> I had a similar problem with my MythTV backend failing to
> unmount /var. It turned out that mythbackend was failing to shutdown
> but openrc carried on trying to shutdown. If I make sure mythbackend
> really is stopped, the reboot proceeds normally, which is much better
> since I had to go into the loft to reboot the box manually,
> 
> 



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] /home doesn't umount on shutdown

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 2091 bytes --]

On Jan 15, 2013 7:59 PM, "Alan McKinnon" <alan.mckinnon@gmail.com> wrote:
>
> On Tue, 15 Jan 2013 11:09:56 +0000
> Neil Bothwick <neil@digimed.co.uk> wrote:
>
> > On Tue, 15 Jan 2013 11:57:21 +0200, Alan McKinnon wrote:
> >
> > > On the rare occasion when I reboot or shut this laptop down, it
> > > continually and consistently gets stuck on one of the final steps,
> > > to umount /home
> >
> > If you logout as your user(s) so only root is logged in, does lsof
> > show any hits for /home?
>
> Only 1 hit - a background ssh process that sets up a bunch of tunnels
> and port forwards so I can get into the corporate network for anywhere.
>
> But I was not able to make the problem re-appear in short reboot
> cycles. So whatever is hanging the box is something that starts up in
> the course of work, it doesn't appear to be there directly after a KDE
> login.
>
> I might have to look at my work flow closely to find all those unusual
> things I do in the course of work. I did manage to see the full
> shutdown output on the screen with these tests though - it
> umounts /home then umounts /boot, but the second umount message is
> never displayed. Seeing as it's just a regular sdb1 partition using
> ext2 I don;t really think it's getting stuck as the next step starts
> up. It's more likely to have something to do with /home
>
>
> >
> > I had a similar problem with my MythTV backend failing to
> > unmount /var. It turned out that mythbackend was failing to shutdown
> > but openrc carried on trying to shutdown. If I make sure mythbackend
> > really is stopped, the reboot proceeds normally, which is much better
> > since I had to go into the loft to reboot the box manually,
> >
> >
>
>
>
> --
> Alan McKinnon
> alan.mckinnon@gmail.com
>
>

A bit roundabout, but you can also try making a 'pseudo-service'. Make it
'depend' on a late-stage service so it starts last, and shuts down early.
The stop() part of the pseudo-service should perform an lsof >> a file (in
a directory still available during the last throes of OpenRC like, say,
/etc).

I hope I'm making sense...

Rgds,

[-- Attachment #2: Type: text/html, Size: 2753 bytes --]

Re: [gentoo-user] /home doesn't umount on shutdown

[Alan McKinnon]

On Tue, 15 Jan 2013 21:47:46 +0700
Pandu Poluan <pandu@poluan.info> wrote:

> On Jan 15, 2013 7:59 PM, "Alan McKinnon" <alan.mckinnon@gmail.com>
> wrote:
> >
> > On Tue, 15 Jan 2013 11:09:56 +0000
> > Neil Bothwick <neil@digimed.co.uk> wrote:
> >
> > > On Tue, 15 Jan 2013 11:57:21 +0200, Alan McKinnon wrote:
> > >
> > > > On the rare occasion when I reboot or shut this laptop down, it
> > > > continually and consistently gets stuck on one of the final
> > > > steps, to umount /home
> > >
> > > If you logout as your user(s) so only root is logged in, does lsof
> > > show any hits for /home?
> >
> > Only 1 hit - a background ssh process that sets up a bunch of
> > tunnels and port forwards so I can get into the corporate network
> > for anywhere.


[snip]

> A bit roundabout, but you can also try making a 'pseudo-service'.
> Make it 'depend' on a late-stage service so it starts last, and shuts
> down early. The stop() part of the pseudo-service should perform an
> lsof >> a file (in a directory still available during the last throes
> of OpenRC like, say, /etc).
> 
> I hope I'm making sense...

Makes perfect sense, a good idea actually :-)

Easiest would be to echo lsof to the console, I only need it if umount
hangs and it will be there and visible. If umount worked it won't be
visible and not needed either


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] /home doesn't umount on shutdown

[Volker Armin Hemmann]

[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]

Fuser is the best way to do IT.
Am 15.01.2013 16:07 schrieb "Alan McKinnon" <alan.mckinnon@gmail.com>:

> On Tue, 15 Jan 2013 21:47:46 +0700
> Pandu Poluan <pandu@poluan.info> wrote:
>
> > On Jan 15, 2013 7:59 PM, "Alan McKinnon" <alan.mckinnon@gmail.com>
> > wrote:
> > >
> > > On Tue, 15 Jan 2013 11:09:56 +0000
> > > Neil Bothwick <neil@digimed.co.uk> wrote:
> > >
> > > > On Tue, 15 Jan 2013 11:57:21 +0200, Alan McKinnon wrote:
> > > >
> > > > > On the rare occasion when I reboot or shut this laptop down, it
> > > > > continually and consistently gets stuck on one of the final
> > > > > steps, to umount /home
> > > >
> > > > If you logout as your user(s) so only root is logged in, does lsof
> > > > show any hits for /home?
> > >
> > > Only 1 hit - a background ssh process that sets up a bunch of
> > > tunnels and port forwards so I can get into the corporate network
> > > for anywhere.
>
>
> [snip]
>
> > A bit roundabout, but you can also try making a 'pseudo-service'.
> > Make it 'depend' on a late-stage service so it starts last, and shuts
> > down early. The stop() part of the pseudo-service should perform an
> > lsof >> a file (in a directory still available during the last throes
> > of OpenRC like, say, /etc).
> >
> > I hope I'm making sense...
>
> Makes perfect sense, a good idea actually :-)
>
> Easiest would be to echo lsof to the console, I only need it if umount
> hangs and it will be there and visible. If umount worked it won't be
> visible and not needed either
>
>
> --
> Alan McKinnon
> alan.mckinnon@gmail.com
>
>
>

[-- Attachment #2: Type: text/html, Size: 2323 bytes --]

[gentoo-user] Re: /home doesn't umount on shutdown

[Remy Blank]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

Alan McKinnon wrote:
> But I was not able to make the problem re-appear in short reboot
> cycles. So whatever is hanging the box is something that starts up in
> the course of work, it doesn't appear to be there directly after a KDE
> login.

I have observed that kded4 and nepomukserver sometimes don't terminate
after logging out of KDE. At the next login, I get another copy of each,
and so they accumulate. It's so bad that I added a script to
~/.kde4/shutdown to "kill -9" them if they are still running 15 seconds
after logging out.

Both processes keep open file descriptors to ~/.xsession-errors, so they
would indeed prevent /home from unmounting.

-- Remy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: /home doesn't umount on shutdown

[Dale]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

Remy Blank wrote:
> Alan McKinnon wrote:
>> But I was not able to make the problem re-appear in short reboot
>> cycles. So whatever is hanging the box is something that starts up in
>> the course of work, it doesn't appear to be there directly after a KDE
>> login.
>
> I have observed that kded4 and nepomukserver sometimes don't terminate
> after logging out of KDE. At the next login, I get another copy of each,
> and so they accumulate. It's so bad that I added a script to
> ~/.kde4/shutdown to "kill -9" them if they are still running 15 seconds
> after logging out.
>
> Both processes keep open file descriptors to ~/.xsession-errors, so they
> would indeed prevent /home from unmounting.
>
> -- Remy
>


I run into that a LOT here too.  After a large upgrade, I usually switch
to the boot runlevel to make sure most everything gets reloaded.  When I
do that, I have to kill some leftover kde stuff manually.  I have been
known to let it sit for several minutes to see if it will eventually
kill itself but it never has.  The nepomukserver is one of them but
there is another one that I can't recall the name of.

May be on to something, hopefully.

Care to share that script?  ;-)  May try that myself.

Dale

:-)  :-)

-- 
I am only responsible for what I said ... Not for what you understood or
how you interpreted my words!


[-- Attachment #2: Type: text/html, Size: 1973 bytes --]

[gentoo-user] Re: /home doesn't umount on shutdown

[Remy Blank]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

Dale wrote:
> Care to share that script?  ;-)  May try that myself.

Sure, here it is:

$ cat .kde4/shutdown/cleanup-kde-processes
#!/bin/bash

log() {
    logger -t "$(basename $0)" "$1"
}

is_running() {
    killall -0 -u $USER "$1"
}

kill_if_running() {
    sleep 15
    is_running startkde && return 0
    if is_running $1; then
        log "$1 still running, sending TERM"
        killall -s TERM $1
        sleep 5
        is_running startkde && return 0
        if is_running $1; then
            log "$1 still running, sending KILL"
            killall -s KILL $1
        fi
    fi
}

kill_if_running kded4 &
kill_if_running nepomukserver &


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Re: /home doesn't umount on shutdown

[Dale]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

Remy Blank wrote:
> Dale wrote:
>> Care to share that script? ;-) May try that myself.
>
> Sure, here it is:
>
> $ cat .kde4/shutdown/cleanup-kde-processes
> #!/bin/bash
>
> log() {
> logger -t "$(basename $0)" "$1"
> }
>
> is_running() {
> killall -0 -u $USER "$1"
> }
>
> kill_if_running() {
> sleep 15
> is_running startkde && return 0
> if is_running $1; then
> log "$1 still running, sending TERM"
> killall -s TERM $1
> sleep 5
> is_running startkde && return 0
> if is_running $1; then
> log "$1 still running, sending KILL"
> killall -s KILL $1
> fi
> fi
> }
>
> kill_if_running kded4 &
> kill_if_running nepomukserver &
>

Thanks much.  This will be better than killing them all one by one.

Dale

:-)  :-)

-- 
I am only responsible for what I said ... Not for what you understood or
how you interpreted my words!


[-- Attachment #2: Type: text/html, Size: 1589 bytes --]

Re: [gentoo-user] Re: /home doesn't umount on shutdown

[Alan McKinnon]

On Tue, 15 Jan 2013 23:37:28 +0100
Remy Blank <remy.blank@pobox.com> wrote:

> Alan McKinnon wrote:
> > But I was not able to make the problem re-appear in short reboot
> > cycles. So whatever is hanging the box is something that starts up
> > in the course of work, it doesn't appear to be there directly after
> > a KDE login.
> 
> I have observed that kded4 and nepomukserver sometimes don't terminate
> after logging out of KDE. At the next login, I get another copy of
> each, and so they accumulate. It's so bad that I added a script to
> ~/.kde4/shutdown to "kill -9" them if they are still running 15
> seconds after logging out.
> 
> Both processes keep open file descriptors to ~/.xsession-errors, so
> they would indeed prevent /home from unmounting.

I recall have similar issues long ago, but haven't seen anything like it
again for months now.

I just checked now and I don't even have kde-base/nepomuk *installed*
anymore. I wonder how that came about, I thought nepomuk was mandatory
for KDE4?

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: /home doesn't umount on shutdown

[Yohan Pereira]

On 16/01/13 at 02:34am, Alan McKinnon wrote:
> I just checked now and I don't even have kde-base/nepomuk *installed*
> anymore. I wonder how that came about, I thought nepomuk was mandatory
> for KDE4?

you probably have USE="-semantic-desktop" :D

-- 

- Yohan Pereira

The difference between a Miracle and a Fact is exactly the difference
between a mermaid and a seal.
                -- Mark Twain

Re: [gentoo-user] Re: /home doesn't umount on shutdown

[Alan McKinnon]

On Wed, 16 Jan 2013 10:46:42 +0530
Yohan Pereira <yohan.pereira@gmail.com> wrote:

> On 16/01/13 at 02:34am, Alan McKinnon wrote:
> > I just checked now and I don't even have kde-base/nepomuk
> > *installed* anymore. I wonder how that came about, I thought
> > nepomuk was mandatory for KDE4?
> 
> you probably have USE="-semantic-desktop" :D
> 

<face palm>

doh, silly me.
Indeed, that's exactly what I have. I blame the 2:34am timestamp.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] /home doesn't umount on shutdown

[Dale]

Alan McKinnon wrote:
> On the rare occasion when I reboot or shut this laptop down, it
> continually and consistently gets stuck on one of the final steps, to
> umount /home
>
> The process never proceeds beyond that point (as /home is always
> fsck'ed on next startup). I can't see any logs as syslog has already
> been shut down at this point, and it happens whether I shutdown as root
> from the console or by using the KDE widget.
>
> /home here is on LVM
>
> I could probably debug this easily enough if I could determine how the
> shutdown sequence is ordered, or get a verbose output. But sadly, my fu
> for such stuff has run out.
>
> Anyone got pointers on where to start poking around?
>
> [I'm not looking for solutions, I'm unlikely to get those right off the
> bat, just looking for pointers atm]
>
>
>

I would do a 'rc single' then use lsof for /home to see what if anything
is still going on.  I think with openrc, when you go to single user it
unmounts about everything, tho this could have changed since it has been
a while since I went to single user.  Oh, I have had issues going from
single user back to default mode.  It just doesn't work right.  So, be
ready to reboot if needed. 

I ran into a weird issue one time a long time ago.  It turned out it was
the order I had them in fstab.  I think I had /usr/portage above /usr so
as it went down the file, it was trying to mount /usr/portage then
trying to mount /usr.  I thought it rather odd, maybe a bug even, but
changing the order made it work.  Do you maybe have something in a odd
order in fstab?

Well, it's early and I am still half asleep.  Hope that helps.  Going
back to bed.  To wet to go hunting this morning, sleeting too.  Brrrrrr!

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!

Re: [gentoo-user] /home doesn't umount on shutdown

[Alan McKinnon]

On Tue, 15 Jan 2013 06:44:11 -0600
Dale <rdalek1967@gmail.com> wrote:

> Alan McKinnon wrote:
> > On the rare occasion when I reboot or shut this laptop down, it
> > continually and consistently gets stuck on one of the final steps,
> > to umount /home
> >
> > The process never proceeds beyond that point (as /home is always
> > fsck'ed on next startup). I can't see any logs as syslog has already
> > been shut down at this point, and it happens whether I shutdown as
> > root from the console or by using the KDE widget.
> >
> > /home here is on LVM
> >
> > I could probably debug this easily enough if I could determine how
> > the shutdown sequence is ordered, or get a verbose output. But
> > sadly, my fu for such stuff has run out.
> >
> > Anyone got pointers on where to start poking around?
> >
> > [I'm not looking for solutions, I'm unlikely to get those right off
> > the bat, just looking for pointers atm]
> >
> >
> >
> 
> I would do a 'rc single' then use lsof for /home to see what if
> anything is still going on.  I think with openrc, when you go to
> single user it unmounts about everything, tho this could have changed
> since it has been a while since I went to single user.  Oh, I have
> had issues going from single user back to default mode.  It just
> doesn't work right.  So, be ready to reboot if needed. 
> 
> I ran into a weird issue one time a long time ago.  It turned out it
> was the order I had them in fstab.  I think I had /usr/portage
> above /usr so as it went down the file, it was trying to
> mount /usr/portage then trying to mount /usr.  I thought it rather
> odd, maybe a bug even, but changing the order made it work.  Do you
> maybe have something in a odd order in fstab?
> 
> Well, it's early and I am still half asleep.  Hope that helps.  Going
> back to bed.  To wet to go hunting this morning, sleeting too.
> Brrrrrr!


LABEL=BOOT              /boot           ext2    noatime         1 2                                 
LABEL=ROOT              /               ext4    noatime,discard 0 1                                 
LABEL=KHAMUL-500G-HOME  /home           ext4    noatime         1 2                                 
                                                                                                    
LABEL=SWAP              swap            swap    sw              0 0                                 
                                                                                                    
/dev/cdrom              /mnt/cdrom      auto    noauto,ro       0 0                                 
/dev/fd0                /mnt/floppy     auto    noauto          0 0                                 
shm                     /dev/shm        tmpfs   size=75%,rw,nosuid,nodev,exec,relatime  0 0         
                                                                                                    
//10.1.249.2/alanm              /mnt/quasar     cifs    noauto,user,credentials=/home/alanm/.credentials/quasar,defaults    0 0                                                                         
172.20.0.3:/mnt/data/media      /mnt/media      nfs     noauto,user,rw,defaults 0 0                 
mtpfs                   /mnt/galaxy     fuse    user,noauto,allow_other         0 0                 


quasar and media are two shares (one at work one at home) that I use all the time, one or both are almost always mounted.
But they don't seem to affect the shutdown at all - /home hangs whether I'm at work or at home and whether the NFS share is accessible or not


> 
> Dale
> 
> :-)  :-) 
> 



-- 
Alan McKinnon
alan.mckinnon@gmail.com