[gentoo-user] Unknown problem causing incorrect mounting and daemons not starting

[gentoo-user] Unknown problem causing incorrect mounting and daemons not starting

[Thomas Sigurdsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

There is an html version of this mail, that also has emerge --info and
my kernel config, at:
http://browniehive.net/tmp/hp-problems.html

I've come to a dead end in troubleshooting this issue and turn to you
for help in how to continue. The following is what I've discovered
during troubleshooting.

# FS's not Mounted After Boot

After login I need to remount `/` with rw. I always then mount `/boot`,
`/mnt/hdd`, `/home/thomas/hdd` manually; that is --- with the `fixmnt`
script I made pasted below.

before doing the above `mount` command tells me that the FS's are
mounted as they should be, but they contain no files.

## `~/bin/fixmnt`

~~~~~~~~~~~~
#!/bin/bash

sudo ip link set lo up
sudo dhclient enp0s25

cp /etc/mtab /home/thomas/mtab-$(date -Iminutes)
wgp=$(wgetpaste /etc/mtab)

sudo mount -o remount,rw /
echo $wgp >> /home/thomas/wgp-$(date -Iminutes)
sudo mount /boot
sudo mount /mnt/hdd
sudo mount /home/thomas/hdd
sudo swapon /dev/sdb2
#ip link set lo up
#sudo ip link set lo up
~~~~~~~~~~~~

As I can't put a copy of `/etc/mtab` anywhere on a read only FS you
won't see what it looks like before running the `fixmnt ` script. The
script gives off an error for the call to cp that I've just ignored for
now (also the wgetpaste doesn't work as it requires creating files in
`/tmp/`, which at that point in time is not writable).

# Daemons Won't Start

You can see from the above that `net.lo` does not get started and there
are some daemons that haven't started and won't start when called. Some
I can start through calling their commands directly (I do this with
`privoxy` and `tor` at the moment in a `tmux` session for browsing).
Other daemons won't start no matter what I try, like `dbus`.

## `/etc/init.d/dbus start`

~~~~~~~~~~~~
* Checking local filesystems  .../dev/sda1 is mounted.
e2fsck: Cannot continue, aborting.


fsck.xfs: invalid option -- 'p'
Usage: fsck.xfs [options] device

Options:
  -f           The device is a file
  -L           Force log zeroing. Do this as a last resort.
  -l logdev    Specifies the device where the external log resides.
  -m maxmem    Maximum amount of memory to be used in megabytes.
  -n           No modify mode, just checks the filesystem for damage.
  -P           Disables prefetching.
  -r rtdev     Specifies the device where the realtime section resides.
  -v           Verbose output.
  -c subopts   Change filesystem parameters - use xfs_admin.
  -o subopts   Override default behaviour, refer to man page.
  -t interval  Reporting interval in seconds.
  -d           Repair dangerously.
  -V           Reports version and exits.
open: No such file or directory
fsck.fat 3.0.28 (2015-05-16)

 * Filesystems couldn't be fixed
                                         [ !! ]
 * ERROR: fsck failed to start
 * Checking local filesystems  .../dev/sda1 is mounted.
e2fsck: Cannot continue, aborting.

.... snip ....

 * Filesystems couldn't be fixed
                                         [ !! ]
 * ERROR: fsck failed to start
 * ERROR: cannot start root as fsck would not start
 * ERROR: cannot start mtab as fsck would not start
 * ERROR: cannot start localmount as fsck would not start
 * ERROR: cannot start dbus as fsck would not start
~~~~~~~~~~~~

Because of the above output I wonder if this is because of some FS or
disk error (crossing fingers for software/config and not hardware) the
culprit could be `fsck.xfs`.

I have run fsck (including `xfs_repair` on the xfs partitions) on all
partitions, with no problems reported from a trisquell livedisk. I also
tried running fsck on `/dev/sda1` from the installed Gentoo instance;
got no errors and didn't change any behaviour as far as I have noticed.

# Overview of System

Sometime in the beginning of October I installed an ssd in this laptop
and migrated to it. The problems started after this, but I am unsure
precisely when as I didn't reboot much in between the migration; for all
I remember now these problems might have started a while after the
migration. `sdb` is the "new" ssd while `sda` is the old hdd.

As you can see from `fstab` I tried putting `/dev/sdb3` as the boot
partition when I installed the ssd.

## `/etc/fstab`

~~~~~~~~~~~~
#/dev/sdb3	/boot		        ext2		noatime		1 2
/dev/sda1	/boot		        ext2		noatime		1 2
/dev/sdb1	/		        ext4		noatime,discard	0 1
/dev/sdb2	none		        swap		sw		0 0
## Old harddisk
/dev/sda4	/mnt/hdd                xfs		noatime		0 2
/mnt/hdd/home/thomas /home/thomas/hdd	none		bind		0 0
~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWayVxAAoJEMUjE08Xv1s5rcoH/3Lz6J7FKByC7xWmkB4iwxx+
oApXUhVfG1+ZJbD6V/EWiPo47aXaen1vOMKadcyw2svpWzZlLpLTQDpOiiGGq9Wj
CCR9VuWlAxJMJZkTm0xEkp7GUAXOR6ln/MJvhbGqnrr1AENj7+QaOVr7rHDziH8M
Ew8oZIQctCFAD8m8dIKkF3E4Jqlu9G7Hyi0MrCuqDJxGrPAXObHKDM2Rlpw1ulb7
bcjekq4Zs0F4VuTpkhM7+6a2jvTCuP1EVdQVGr/J3Czz2P1X6sZr9Wbd9o9TvQO/
LWmYT2KaYXPcWcx3LfWw7U51pv6b5Qf0MS87eEMtcnXjNaIDGt3ADzjMpYJ5ES0=
=FlkE
-----END PGP SIGNATURE-----

Re: [gentoo-user] Unknown problem causing incorrect mounting and daemons not starting

[John Runyon]

It seems to me the root of your problems is fsck failing because (it thinks) the partitions are already mounted.

Is your mtab a symlink to /proc/mounts as it "should" be?

Thomas Sigurdsen <thomas.sigurdsen@gmail.com> wrote:

>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA256
>
>There is an html version of this mail, that also has emerge --info and
>my kernel config, at:
>http://browniehive.net/tmp/hp-problems.html
>
>I've come to a dead end in troubleshooting this issue and turn to you
>for help in how to continue. The following is what I've discovered
>during troubleshooting.
>
># FS's not Mounted After Boot
>
>After login I need to remount `/` with rw. I always then mount `/boot`,
>`/mnt/hdd`, `/home/thomas/hdd` manually; that is --- with the `fixmnt`
>script I made pasted below.
>
>before doing the above `mount` command tells me that the FS's are
>mounted as they should be, but they contain no files.
>
>## `~/bin/fixmnt`
>
>~~~~~~~~~~~~
>#!/bin/bash
>
>sudo ip link set lo up
>sudo dhclient enp0s25
>
>cp /etc/mtab /home/thomas/mtab-$(date -Iminutes)
>wgp=$(wgetpaste /etc/mtab)
>
>sudo mount -o remount,rw /
>echo $wgp >> /home/thomas/wgp-$(date -Iminutes)
>sudo mount /boot
>sudo mount /mnt/hdd
>sudo mount /home/thomas/hdd
>sudo swapon /dev/sdb2
>#ip link set lo up
>#sudo ip link set lo up
>~~~~~~~~~~~~
>
>As I can't put a copy of `/etc/mtab` anywhere on a read only FS you
>won't see what it looks like before running the `fixmnt ` script. The
>script gives off an error for the call to cp that I've just ignored for
>now (also the wgetpaste doesn't work as it requires creating files in
>`/tmp/`, which at that point in time is not writable).
>
># Daemons Won't Start
>
>You can see from the above that `net.lo` does not get started and there
>are some daemons that haven't started and won't start when called. Some
>I can start through calling their commands directly (I do this with
>`privoxy` and `tor` at the moment in a `tmux` session for browsing).
>Other daemons won't start no matter what I try, like `dbus`.
>
>## `/etc/init.d/dbus start`
>
>~~~~~~~~~~~~
>* Checking local filesystems  .../dev/sda1 is mounted.
>e2fsck: Cannot continue, aborting.
>
>
>fsck.xfs: invalid option -- 'p'
>Usage: fsck.xfs [options] device
>
>Options:
>  -f           The device is a file
>  -L           Force log zeroing. Do this as a last resort.
>  -l logdev    Specifies the device where the external log resides.
>  -m maxmem    Maximum amount of memory to be used in megabytes.
>  -n           No modify mode, just checks the filesystem for damage.
>  -P           Disables prefetching.
>  -r rtdev     Specifies the device where the realtime section resides.
>  -v           Verbose output.
>  -c subopts   Change filesystem parameters - use xfs_admin.
>  -o subopts   Override default behaviour, refer to man page.
>  -t interval  Reporting interval in seconds.
>  -d           Repair dangerously.
>  -V           Reports version and exits.
>open: No such file or directory
>fsck.fat 3.0.28 (2015-05-16)
>
> * Filesystems couldn't be fixed
>                                         [ !! ]
> * ERROR: fsck failed to start
> * Checking local filesystems  .../dev/sda1 is mounted.
>e2fsck: Cannot continue, aborting.
>
>.... snip ....
>
> * Filesystems couldn't be fixed
>                                         [ !! ]
> * ERROR: fsck failed to start
> * ERROR: cannot start root as fsck would not start
> * ERROR: cannot start mtab as fsck would not start
> * ERROR: cannot start localmount as fsck would not start
> * ERROR: cannot start dbus as fsck would not start
>~~~~~~~~~~~~
>
>Because of the above output I wonder if this is because of some FS or
>disk error (crossing fingers for software/config and not hardware) the
>culprit could be `fsck.xfs`.
>
>I have run fsck (including `xfs_repair` on the xfs partitions) on all
>partitions, with no problems reported from a trisquell livedisk. I also
>tried running fsck on `/dev/sda1` from the installed Gentoo instance;
>got no errors and didn't change any behaviour as far as I have noticed.
>
># Overview of System
>
>Sometime in the beginning of October I installed an ssd in this laptop
>and migrated to it. The problems started after this, but I am unsure
>precisely when as I didn't reboot much in between the migration; for all
>I remember now these problems might have started a while after the
>migration. `sdb` is the "new" ssd while `sda` is the old hdd.
>
>As you can see from `fstab` I tried putting `/dev/sdb3` as the boot
>partition when I installed the ssd.
>
>## `/etc/fstab`
>
>~~~~~~~~~~~~
>#/dev/sdb3	/boot		        ext2		noatime		1 2
>/dev/sda1	/boot		        ext2		noatime		1 2
>/dev/sdb1	/		        ext4		noatime,discard	0 1
>/dev/sdb2	none		        swap		sw		0 0
>## Old harddisk
>/dev/sda4	/mnt/hdd                xfs		noatime		0 2
>/mnt/hdd/home/thomas /home/thomas/hdd	none		bind		0 0
>~~~~~~~~~~~~
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v2
>
>iQEcBAEBCAAGBQJWayVxAAoJEMUjE08Xv1s5rcoH/3Lz6J7FKByC7xWmkB4iwxx+
>oApXUhVfG1+ZJbD6V/EWiPo47aXaen1vOMKadcyw2svpWzZlLpLTQDpOiiGGq9Wj
>CCR9VuWlAxJMJZkTm0xEkp7GUAXOR6ln/MJvhbGqnrr1AENj7+QaOVr7rHDziH8M
>Ew8oZIQctCFAD8m8dIKkF3E4Jqlu9G7Hyi0MrCuqDJxGrPAXObHKDM2Rlpw1ulb7
>bcjekq4Zs0F4VuTpkhM7+6a2jvTCuP1EVdQVGr/J3Czz2P1X6sZr9Wbd9o9TvQO/
>LWmYT2KaYXPcWcx3LfWw7U51pv6b5Qf0MS87eEMtcnXjNaIDGt3ADzjMpYJ5ES0=
>=FlkE
>-----END PGP SIGNATURE-----
>

Re: [gentoo-user] Unknown problem causing incorrect mounting and daemons not starting

[Thomas Sigurdsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

> It seems to me the root of your problems is fsck failing because (it
> thinks) the partitions are already mounted.
> 
> Is your mtab a symlink to /proc/mounts as it "should" be?

Just checked, and mtab is not a symlink. It is a plain text file.

Just tried takinga backup of mtab and symlinking /proc/mounts onto
/etc/mtab yields no different results when trying `/etc/init.d/dbus
start`. Is this something I fix somewhere else or maybe I just need to
reboot the system after symlinking.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWbIDrAAoJEMUjE08Xv1s5xiMH/RHhiWklaZiMes7h5PSVLP7M
JLJJ14jpqytxPfDujZfUznT+Yed/QxvwNSG2lK7YidxKxrTwB/F8cslFKHulDMc/
Sp4knftRAdHu9f21lkt+LWgfryu6ZxOu4lzCjGev/47yFU7nOb/5BWfBVinGm8vo
lZVweuEQJOmqs1k7cl5EpYMbKzx6YYuQpkGddV5VK9Gq99r6zSCThpc/XkS49gLF
DTblOWBXcJcmw3vWwsYHwdGz2eIAdCN5LJ0brRCwocWneZUXRouDuITBhGCtWL1W
gnSp6W/WWY9khINvZmiJUxxP0Oe+YQ+88MBLly2f4ptn4NUVJFA+gRm6XOlw/50=
=Wsr+
-----END PGP SIGNATURE-----

Re: [gentoo-user] Unknown problem causing incorrect mounting and daemons not starting

[John Runyon]

I wouldn't expect the symlink to fix anything without a reboot (since the main problem is at startup time). I'm not sure if the dbus issue is separate or not - if it is the same it seems odd that it won't start even after getting everything mounted by hand, anyway.

-John
Sent from my phone

Thomas Sigurdsen <thomas.sigurdsen@gmail.com> wrote:

>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA256
>
>> It seems to me the root of your problems is fsck failing because (it
>> thinks) the partitions are already mounted.
>> 
>> Is your mtab a symlink to /proc/mounts as it "should" be?
>
>Just checked, and mtab is not a symlink. It is a plain text file.
>
>Just tried takinga backup of mtab and symlinking /proc/mounts onto
>/etc/mtab yields no different results when trying `/etc/init.d/dbus
>start`. Is this something I fix somewhere else or maybe I just need to
>reboot the system after symlinking.
>
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v2
>
>iQEcBAEBCAAGBQJWbIDrAAoJEMUjE08Xv1s5xiMH/RHhiWklaZiMes7h5PSVLP7M
>JLJJ14jpqytxPfDujZfUznT+Yed/QxvwNSG2lK7YidxKxrTwB/F8cslFKHulDMc/
>Sp4knftRAdHu9f21lkt+LWgfryu6ZxOu4lzCjGev/47yFU7nOb/5BWfBVinGm8vo
>lZVweuEQJOmqs1k7cl5EpYMbKzx6YYuQpkGddV5VK9Gq99r6zSCThpc/XkS49gLF
>DTblOWBXcJcmw3vWwsYHwdGz2eIAdCN5LJ0brRCwocWneZUXRouDuITBhGCtWL1W
>gnSp6W/WWY9khINvZmiJUxxP0Oe+YQ+88MBLly2f4ptn4NUVJFA+gRm6XOlw/50=
>=Wsr+
>-----END PGP SIGNATURE-----
>

Re: [gentoo-user] Unknown problem causing incorrect mounting and daemons not starting

[Thomas Sigurdsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

The problems/strange state persists with mtab being a symlink and then
rebooting. Mtab stays a symlink through reboots too.

> I wouldn't expect the symlink to fix anything without a reboot (since
> the main problem is at startup time). I'm not sure if the dbus issue
> is separate or not - if it is the same it seems odd that it won't
> start even after getting everything mounted by hand, anyway.
> 
> -John Sent from my phone
> 
> Thomas Sigurdsen <thomas.sigurdsen@gmail.com> wrote:
> 
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>> 
>>> It seems to me the root of your problems is fsck failing because
>>> (it thinks) the partitions are already mounted.
>>> 
>>> Is your mtab a symlink to /proc/mounts as it "should" be?
>> 
>> Just checked, and mtab is not a symlink. It is a plain text file.
>> 
>> Just tried takinga backup of mtab and symlinking /proc/mounts onto 
>> /etc/mtab yields no different results when trying
>> `/etc/init.d/dbus start`. Is this something I fix somewhere else or
>> maybe I just need to reboot the system after symlinking.
>> 
>> -----BEGIN PGP SIGNATURE----- Version: GnuPG v2
>> 
>> iQEcBAEBCAAGBQJWbIDrAAoJEMUjE08Xv1s5xiMH/RHhiWklaZiMes7h5PSVLP7M 
>> JLJJ14jpqytxPfDujZfUznT+Yed/QxvwNSG2lK7YidxKxrTwB/F8cslFKHulDMc/ 
>> Sp4knftRAdHu9f21lkt+LWgfryu6ZxOu4lzCjGev/47yFU7nOb/5BWfBVinGm8vo 
>> lZVweuEQJOmqs1k7cl5EpYMbKzx6YYuQpkGddV5VK9Gq99r6zSCThpc/XkS49gLF 
>> DTblOWBXcJcmw3vWwsYHwdGz2eIAdCN5LJ0brRCwocWneZUXRouDuITBhGCtWL1W 
>> gnSp6W/WWY9khINvZmiJUxxP0Oe+YQ+88MBLly2f4ptn4NUVJFA+gRm6XOlw/50= 
>> =Wsr+ -----END PGP SIGNATURE-----
>> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJWc+axAAoJEMUjE08Xv1s5y+oH/0lrOLDp7bNWQWgvIW2JN8as
y+qxKPvW4WCXg0677qB/PCVcwPwdhQlKeSp2D1Qbip7pJTqoNYnTmpuIW41BQjNZ
o4/8HV4LuWNUJBIVT45h7RODkLnRy1ewgd1z4PbQjAORLuCElGR3/XXa/G28Pi5j
QTz9yvauPyZEAxrOARTDRLj3Q9ZVTMBR3cOXJ3vnemBmxkE8Di+9EO6ev5N2i4U7
kbIyXU6DmFgco5XPWtNn84biNrlEue3O4vz/VL9j62yuJ/B2l+YK02ocuTXhJfZY
pAvYfDF6FzeWcNmfO4PW6TPP4EV4WVT8amTiMUdlzQXW7eMdb2S0RlAtYYTEbvg=
=lb1j
-----END PGP SIGNATURE-----

Re: [gentoo-user] Unknown problem causing incorrect mounting and daemons not starting

[Alan Mackenzie]

Hello, Thomas.

On Fri, Dec 18, 2015 at 10:57:53AM +0000, Thomas Sigurdsen wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256

> The problems/strange state persists with mtab being a symlink and then
> rebooting. Mtab stays a symlink through reboots too.

This is a stab in the dark, but recently when an emerge upgraded udev, I
got a warning message that CONFIG_FHANDLE wasn't set in my kernel config.
(This is in General setup, prompt "open by fhandle syscalls".)

Without this setting, I got errors on booting, and indeed my ethernet
connection didn't come up.  After setting CONFIG_FHANDLE and rebuilding
my kernel, the problem went away.

Might you have this same problem?

-- 
Alan Mackenzie (Nuremberg, Germany).