[gentoo-user] Strange new behavior from the "mount" command

[gentoo-user] Strange new behavior from the "mount" command

[walt]

I have two similar but not identical ~amd64 machines, and *one* of the
two machines is doing something new and strange when I type "mount" with
no arguments.

The "bad" machine prints the list of mounted filesystems as it should,
but then proceeds to read the partition table on every disk in the machine
and writes a fresh version of /run/blkid/blkid.tab .

This has the very annoying side effect of spinning up any sleeping disks,
including the floppy disk (but not the dvd player, thankfully).

I re-installed util-linux, which installs the "mount" utility, but no
difference.  (The two machines both have util-linux-2.26.1-r1).

This new behavior began on April 14, FWIW, and the only package I installed
on that machine that day was gentoo-sources-3.14.38, which is why I blamed
the new kernel for the new behavior but I discovered since then that it
happens with all the old kernels too.

I'm stumped.  Any ideas?

Re: [gentoo-user] Strange new behavior from the "mount" command

[Paul Colquhoun]

[-- Attachment #1: Type: text/plain, Size: 1923 bytes --]

On Sat, 18 Apr 2015 11:48:12 walt wrote:
> I have two similar but not identical ~amd64 machines, and *one* of the
> two machines is doing something new and strange when I type "mount" 
with
> no arguments.
> 
> The "bad" machine prints the list of mounted filesystems as it should,
> but then proceeds to read the partition table on every disk in the 
machine
> and writes a fresh version of /run/blkid/blkid.tab .
> 
> This has the very annoying side effect of spinning up any sleeping disks,
> including the floppy disk (but not the dvd player, thankfully).
> 
> I re-installed util-linux, which installs the "mount" utility, but no
> difference.  (The two machines both have util-linux-2.26.1-r1).
> 
> This new behavior began on April 14, FWIW, and the only package I 
installed
> on that machine that day was gentoo-sources-3.14.38, which is why I 
blamed
> the new kernel for the new behavior but I discovered since then that it
> happens with all the old kernels too.
> 
> I'm stumped.  Any ideas?


Are you sure they are both running the same mount command?

What does 'type mount' or 'which mount' show for each machine?

Is the 'bad' machine perhaps using the '-l' option, which looks like it may 
need to read information from partitions on the fly:

-l, --show-labels
    Add the labels in the mount output.  mount must have permission to 
read
    the disk device (e.g. be suid root) for this to work.  One can set such
    a label for ext2, ext3 or ext4 using the e2label(8) utility, or for XFS
    using xfs_admin(8), or for reiserfs using reiserfstune(8).

On the other hand, using '-l' on my machine didn't appear to try anything, 
and didn't rewrite /run/blkid/blkid.tab but that may be because I don't use 
labels.


-- 
Reverend Paul Colquhoun, ULC.     http://andor.dropbear.id.au/
  Asking for technical help in newsgroups?  Read this first:
     http://catb.org/~esr/faqs/smart-questions.html#intro


[-- Attachment #2: Type: text/html, Size: 8516 bytes --]

[gentoo-user] Re: Strange new behavior from the "mount" command

[walt]

On 04/18/2015 02:31 PM, Paul Colquhoun wrote:
> On Sat, 18 Apr 2015 11:48:12 walt wrote:
> 
>> I have two similar but not identical ~amd64 machines, and *one* of
>> the
> 
>> two machines is doing something new and strange when I type "mount"
>> with
> 
>> no arguments.
> 
>> 
> 
>> The "bad" machine prints the list of mounted filesystems as it
>> should,
> 
>> but then proceeds to read the partition table on every disk in the
>> machine
> 
>> and writes a fresh version of /run/blkid/blkid.tab .
> 
>> 
> 
>> This has the very annoying side effect of spinning up any sleeping
>> disks,
> 
>> including the floppy disk (but not the dvd player, thankfully).
> 
>> 
> 
>> I re-installed util-linux, which installs the "mount" utility, but
>> no
> 
>> difference. (The two machines both have util-linux-2.26.1-r1).
> 
>> 
> 
>> This new behavior began on April 14, FWIW, and the only package I
>> installed
> 
>> on that machine that day was gentoo-sources-3.14.38, which is why I
>> blamed
> 
>> the new kernel for the new behavior but I discovered since then
>> that it
> 
>> happens with all the old kernels too.
> 
>> 
> 
>> I'm stumped. Any ideas?
> 
> 
> 
> 
> 
> Are you sure they are both running the same mount command?
> 
> 
> 
> What does 'type mount' or 'which mount' show for each machine?
> 
> 
> 
> Is the 'bad' machine perhaps using the '-l' option, which looks like
> it may need to read information from partitions on the fly:
> 
> 
> 
> -l, --show-labels
> 
> Add the labels in the mount output. mount must have permission to
> read
> 
> the disk device (e.g. be suid root) for this to work. One can set
> such
> 
> a label for ext2, ext3 or ext4 using the e2label(8) utility, or for
> XFS
> 
> using xfs_admin(8), or for reiserfs using reiserfstune(8).
> 
> 
> 
> On the other hand, using '-l' on my machine didn't appear to try
> anything, and didn't rewrite /run/blkid/blkid.tab but that may be
> because I don't use labels.

Good questions, thanks.

I know both machines are actually running /bin/mount and not using any
arguments (like -l) because strace shows me that info in its first line
of output:

execve("/bin/mount", ["mount"], [/* 61 vars */]) = 0

That number 61 on the 'bad' machine is 48, though, and I don't know where
that odd-looking string of characters is generated or what it means. To me
it looks like a comment in a file of 'c' code.

Still stumped :(

Re: [gentoo-user] Re: Strange new behavior from the "mount" command

[Fernando Rodriguez]

On Saturday, April 18, 2015 3:59:15 PM walt wrote:
> 
> execve("/bin/mount", ["mount"], [/* 61 vars */]) = 0
> 
> That number 61 on the 'bad' machine is 48, though, and I don't know where
> that odd-looking string of characters is generated or what it means. To me
> it looks like a comment in a file of 'c' code.
> 
> Still stumped :(

That would be the number of environment variables passed to execve. strace is 
just trying not to be too noisy.


Are there any differences in the options used in fstab between both machines, 
Especially the auto or noauto options or if one of them is using labels. The 
mount(8) man page may have more hints.

-- 
Fernando Rodriguez

[gentoo-user] Re: Strange new behavior from the "mount" command

[walt]

On 04/18/2015 05:33 PM, Fernando Rodriguez wrote:
> On Saturday, April 18, 2015 3:59:15 PM walt wrote:
>>
>> execve("/bin/mount", ["mount"], [/* 61 vars */]) = 0
>>
>> That number 61 on the 'bad' machine is 48, though, and I don't know where
>> that odd-looking string of characters is generated or what it means. To me
>> it looks like a comment in a file of 'c' code.
>>
>> Still stumped :(
> 
> That would be the number of environment variables passed to execve. strace is 
> just trying not to be too noisy.
> 
> 
> Are there any differences in the options used in fstab between both machines, 
> Especially the auto or noauto options or if one of them is using labels. The 
> mount(8) man page may have more hints.

As a quick-and-dirty way of testing your idea I moved /etc/fstab out of the way.

I was surprised to learn that "mount" doesn't care about fstab, and doesn't even
bother to look for it (when invoked with no arguments).

Using strace I examined the list of files opened by "mount" on both machines and
discovered that the "bad" machine reads the same 8 files as the "good" machine,
but then proceeds to read /etc/blkid.conf (I don't have that file on either machine)
and then /proc/cmdline.  (And then proceeds to read all the partition tables, etc)

So, what kind of process needs those two files to do its job?  I have no idea, but
I'm hoping that some of you smart people out there may know.

Re: [gentoo-user] Re: Strange new behavior from the "mount" command

[Fernando Rodriguez]

On Sunday, April 19, 2015 3:18:38 PM walt wrote:
> 
> As a quick-and-dirty way of testing your idea I moved /etc/fstab out of the 
way.
> 
> I was surprised to learn that "mount" doesn't care about fstab, and doesn't 
even
> bother to look for it (when invoked with no arguments).
> 

You'll have to play with the options to test my idea. It doesn't need to 
access it when called without arguments but it does (either directly or 
indirectly) when mounting volumes, either by mount -a at boot or whatever was 
used to mount them. Since mount(8) is an interface to mount(2) and friends, 
all the mount options are already cached by the kernel and it'll have access 
to them. You can also tell if the mount options on fstab are overriden by 
something else by comparing the output of mount (with no args) with your fstab 
file.

-- 
Fernando Rodriguez

Re: [gentoo-user] Re: Strange new behavior from the "mount" command

[Mike Gilbert]

On Sun, Apr 19, 2015 at 6:18 PM, walt <w41ter@gmail.com> wrote:
> On 04/18/2015 05:33 PM, Fernando Rodriguez wrote:
>> On Saturday, April 18, 2015 3:59:15 PM walt wrote:
>>>
>>> execve("/bin/mount", ["mount"], [/* 61 vars */]) = 0
>>>
>>> That number 61 on the 'bad' machine is 48, though, and I don't know where
>>> that odd-looking string of characters is generated or what it means. To me
>>> it looks like a comment in a file of 'c' code.
>>>
>>> Still stumped :(
>>
>> That would be the number of environment variables passed to execve. strace is
>> just trying not to be too noisy.
>>
>>
>> Are there any differences in the options used in fstab between both machines,
>> Especially the auto or noauto options or if one of them is using labels. The
>> mount(8) man page may have more hints.
>
> As a quick-and-dirty way of testing your idea I moved /etc/fstab out of the way.
>
> I was surprised to learn that "mount" doesn't care about fstab, and doesn't even
> bother to look for it (when invoked with no arguments).
>

It reads information from /etc/mtab primarily, as well as
/run/mount/utab. Also, if /etc/mtab is a symlink, it reads from
/proc/self/mountinfo instead of /etc/mtab.

It seems like there is probably some difference in the data it is
reading from those files on your system. Maybe post them so we can all
have a look?

[gentoo-user] Re: Strange new behavior from the "mount" command

[walt]

On 04/19/2015 05:45 PM, Mike Gilbert wrote:
> On Sun, Apr 19, 2015 at 6:18 PM, walt <w41ter@gmail.com> wrote:

>> As a quick-and-dirty way of testing your idea I moved /etc/fstab out of the way.
>>
>> I was surprised to learn that "mount" doesn't care about fstab, and doesn't even
>> bother to look for it (when invoked with no arguments).
>>
> 
> It reads information from /etc/mtab primarily, as well as
> /run/mount/utab. Also, if /etc/mtab is a symlink, it reads from
> /proc/self/mountinfo instead of /etc/mtab.
> 
> It seems like there is probably some difference in the data it is
> reading from those files on your system. Maybe post them so we can all
> have a look?

I really appreciate your help, thanks.  Sorry there's so much to read through.

I avoided the possible caching problem Francisco mentioned by booting the machine
without an /etc/fstab, so it wound up with only / and swap partitions actually
mounted.

Here are the files that "mount" opened (running with no arguments) that it
normally would not open:

#cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt5)/boot/vmlinuz root=PARTUUID=345FD3C4-9E1C-49EB-859C-E3A3034325B3 rootwait init=/usr/lib64/systemd/systemd

#cat /proc/partitions (only sdb5 and sdb6 are actually mounted atm)
major minor  #blocks  name

   8        0  120060864 sda
   8        1       6144 sda1
   8        5   15253654 sda5
   8        6   27455053 sda6
   8        7   25679871 sda7
   8        8   25832488 sda8
   8        9   25832488 sda9
  11        0    1048575 sr0
   8       16   78150744 sdb
   8       17    2313328 sdb1
   8       18       2048 sdb2
   8       21   20482843 sdb5
   8       22    3092481 sdb6
   8       23   26129691 sdb7
   8       24   26129691 sdb8
   2        0          4 fd0
   8       32  488386584 sdc
   8       33          1 sdc1
   8       34    4192933 sdc2
   8       35       7168 sdc3
   8       37  484185112 sdc5
 254        0  157044736 dm-0

/etc/blkid.conf (but I don't have one)

#cat /run/blkid/blkid.tab
<device DEVNO="0x0805" TIME="1429555704.829004" LABEL="amd64" UUID="e976fc30-c08f-6a5b-18c7-ef57278a08ad" TYPE="ext4" PARTLABEL="Linux/Windows data" PARTUUID="b098397c-68c3-4b56-bd15-6364c3c2108e">/dev/sda5</device>
<device DEVNO="0x0806" TIME="1429555704.829387" UUID="TMWa80-CXcZ-iQkR-SkQY-0xPe-p834-iNki91" TYPE="LVM2_member" PARTLABEL="Linux LVM" PARTUUID="68083bec-62f4-4979-8cd8-377d0cb62ac1">/dev/sda6</device>
<device DEVNO="0x0807" TIME="1429555704.829692" UUID="ehtwkq-Z4LH-Pdjn-5vp3-xRhM-12ho-OjlhZ9" TYPE="LVM2_member" PARTLABEL="Linux LVM" PARTUUID="b9b2a14b-b983-4cad-ac5f-cc1f5aad0a28">/dev/sda7</device>
<device DEVNO="0x0808" TIME="1429555704.829969" UUID="gyDXEl-y1Xd-Yll1-rtcf-I3ph-t22r-xk9r21" TYPE="LVM2_member" PARTLABEL="Linux LVM" PARTUUID="08c8a3a0-dabd-4921-980e-1b5bca87f258">/dev/sda8</device>
<device DEVNO="0x0809" TIME="1429555704.830247" UUID="JnKcRI-WGb4-SymA-3xi8-3YOU-8VaQ-YK4U75" TYPE="LVM2_member" PARTLABEL="Linux LVM" PARTUUID="fcc56ca1-a76c-44fb-a42c-76fe10d56120">/dev/sda9</device>
<device DEVNO="0x0811" TIME="1429555704.970765" LABEL="C" UUID="3D15-CC2D" TYPE="vfat" PARTLABEL="Linux/Windows data" PARTUUID="71dd1824-2a9a-4f17-afa1-32c9bfa5cab6">/dev/sdb1</device>
<device DEVNO="0x0815" TIME="1429555704.971519" LABEL="gentoo64unstable" UUID="b2cce2df-05cd-0d3a-9fdd-30d4b4ceffce" TYPE="ext4" PARTLABEL="Linux/Windows data" PARTUUID="345fd3c4-9e1c-49eb-859c-e3a3034325b3">/dev/sdb5</device>
<device DEVNO="0x0816" TIME="1429555704.972341" LABEL="SWAP" UUID="f09da9c9-968a-4a33-a01d-89ee41c5fa47" TYPE="swap" PARTLABEL="Linux swap" PARTUUID="bb618fd6-6be9-4bb4-b3aa-73a81c47edbd">/dev/sdb6</device>
<device DEVNO="0x0817" TIME="1429555704.972706" UUID="p8Dc0Z-eBoA-8F5q-ehxl-p0Ce-p2Z2-Q8ukuU" TYPE="LVM2_member" PARTLABEL="Linux LVM" PARTUUID="b9f85a57-2bbc-48ff-baf7-2dbd5d10e573">/dev/sdb7</device>
<device DEVNO="0x0818" TIME="1429555704.972988" UUID="fJv0XT-3Ge2-Ipm4-y20s-wOv0-cClc-qcBO59" TYPE="LVM2_member" PARTLABEL="Linux LVM" PARTUUID="40a60278-daec-43a1-95e1-07308600467f">/dev/sdb8</device>
<device DEVNO="0x0822" TIME="1429555705.7818" LABEL="FAT32SAM" UUID="B4F6-C857" TYPE="vfat" PARTUUID="000c4056-02">/dev/sdc2</device>
<device DEVNO="0x0825" TIME="1429555705.18210" LABEL="SAMSUNG" UUID="f8c60d37-f86a-4117-b71e-546de21cbd16" TYPE="ext4" PARTUUID="000c4056-05">/dev/sdc5</device>
<device DEVNO="0xfe00" TIME="1429555705.123208" PRI="45" LABEL="U" UUID="482ec9a2-55cb-4f9d-8fff-ce73e29c1864" SEC_TYPE="ext2" TYPE="ext3">/dev/mapper/vg0-u</device>

/proc/evms/volumes (but I don't have one)

And (because /etc/mtab is a symlink to /proc/self/mounts):
#cat /proc/self/mounts
rootfs / rootfs rw 0 0
/dev/root / ext4 rw,relatime,data=ordered 0 0
devtmpfs /dev devtmpfs rw,relatime,size=1929384k,nr_inodes=482346,mode=755 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,nosuid,nodev,noexec,relatime,cpu 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
tmpfs /tmp tmpfs rw 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
tmpfs /run/user/0 tmpfs rw,nosuid,nodev,relatime,size=392552k,mode=700 0 0

#cat /proc/self/mountinfo
12 0 8:21 / / rw,relatime shared:1 - ext4 /dev/root rw,data=ordered
13 12 0:5 / /dev rw,relatime shared:2 - devtmpfs devtmpfs rw,size=1929384k,nr_inodes=482346,mode=755
14 12 0:12 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw
15 12 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:8 - proc proc rw
16 13 0:13 / /dev/shm rw,nosuid,nodev shared:3 - tmpfs tmpfs rw
17 13 0:10 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
18 12 0:14 / /run rw,nosuid,nodev shared:9 - tmpfs tmpfs rw,mode=755
19 14 0:15 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:6 - tmpfs tmpfs ro,mode=755
20 19 0:16 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:7 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
21 19 0:17 / /sys/fs/cgroup/cpu rw,nosuid,nodev,noexec,relatime shared:10 - cgroup cgroup rw,cpu
22 15 0:18 / /proc/sys/fs/binfmt_misc rw,relatime shared:11 - autofs systemd-1 rw,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct
23 12 0:19 / /var/lib/nfs/rpc_pipefs rw,relatime shared:12 - rpc_pipefs sunrpc rw
24 13 0:11 / /dev/mqueue rw,relatime shared:13 - mqueue mqueue rw
25 12 0:20 / /tmp rw shared:14 - tmpfs tmpfs rw
65 22 0:22 / /proc/sys/fs/binfmt_misc rw,relatime shared:37 - binfmt_misc binfmt_misc rw
62 18 0:21 / /run/user/0 rw,nosuid,nodev,relatime shared:35 - tmpfs tmpfs rw,size=392552k,mode=700


Again, many thanks!

Re: [gentoo-user] Re: Strange new behavior from the "mount" command

[Mike Gilbert]

On Mon, Apr 20, 2015 at 3:21 PM, walt <w41ter@gmail.com> wrote:
> On 04/19/2015 05:45 PM, Mike Gilbert wrote:
>> On Sun, Apr 19, 2015 at 6:18 PM, walt <w41ter@gmail.com> wrote:
>
>>> As a quick-and-dirty way of testing your idea I moved /etc/fstab out of the way.
>>>
>>> I was surprised to learn that "mount" doesn't care about fstab, and doesn't even
>>> bother to look for it (when invoked with no arguments).
>>>
>>
>> It reads information from /etc/mtab primarily, as well as
>> /run/mount/utab. Also, if /etc/mtab is a symlink, it reads from
>> /proc/self/mountinfo instead of /etc/mtab.
>>
>> It seems like there is probably some difference in the data it is
>> reading from those files on your system. Maybe post them so we can all
>> have a look?
>
> I really appreciate your help, thanks.  Sorry there's so much to read through.
>
> I avoided the possible caching problem Francisco mentioned by booting the machine
> without an /etc/fstab, so it wound up with only / and swap partitions actually
> mounted.
>
> Here are the files that "mount" opened (running with no arguments) that it
> normally would not open:
>
> #cat /proc/cmdline
> BOOT_IMAGE=(hd0,gpt5)/boot/vmlinuz root=PARTUUID=345FD3C4-9E1C-49EB-859C-E3A3034325B3 rootwait init=/usr/lib64/systemd/systemd
>
> #cat /proc/self/mountinfo
> 12 0 8:21 / / rw,relatime shared:1 - ext4 /dev/root rw,data=ordered

I think this may be related to having /dev/root appear in
/proc/self/moutinfo. In this case, mount will look for your root
filesystem in /proc/cmdline, and resolve it from there.

Since your kernel command line has a PARTUUID tag, it probably needs
to scan the partition tables to resolve that. This is mostly a SWAG; I
didn't trace the code to this point.

Re: [gentoo-user] Re: Strange new behavior from the "mount" command

[Mike Gilbert]

On Mon, Apr 20, 2015 at 4:01 PM, Mike Gilbert <floppym@gentoo.org> wrote:
> On Mon, Apr 20, 2015 at 3:21 PM, walt <w41ter@gmail.com> wrote:
>> On 04/19/2015 05:45 PM, Mike Gilbert wrote:
>>> On Sun, Apr 19, 2015 at 6:18 PM, walt <w41ter@gmail.com> wrote:
>>
>>>> As a quick-and-dirty way of testing your idea I moved /etc/fstab out of the way.
>>>>
>>>> I was surprised to learn that "mount" doesn't care about fstab, and doesn't even
>>>> bother to look for it (when invoked with no arguments).
>>>>
>>>
>>> It reads information from /etc/mtab primarily, as well as
>>> /run/mount/utab. Also, if /etc/mtab is a symlink, it reads from
>>> /proc/self/mountinfo instead of /etc/mtab.
>>>
>>> It seems like there is probably some difference in the data it is
>>> reading from those files on your system. Maybe post them so we can all
>>> have a look?
>>
>> I really appreciate your help, thanks.  Sorry there's so much to read through.
>>
>> I avoided the possible caching problem Francisco mentioned by booting the machine
>> without an /etc/fstab, so it wound up with only / and swap partitions actually
>> mounted.
>>
>> Here are the files that "mount" opened (running with no arguments) that it
>> normally would not open:
>>
>> #cat /proc/cmdline
>> BOOT_IMAGE=(hd0,gpt5)/boot/vmlinuz root=PARTUUID=345FD3C4-9E1C-49EB-859C-E3A3034325B3 rootwait init=/usr/lib64/systemd/systemd
>>
>> #cat /proc/self/mountinfo
>> 12 0 8:21 / / rw,relatime shared:1 - ext4 /dev/root rw,data=ordered
>
> I think this may be related to having /dev/root appear in
> /proc/self/moutinfo. In this case, mount will look for your root
> filesystem in /proc/cmdline, and resolve it from there.
>
> Since your kernel command line has a PARTUUID tag, it probably needs
> to scan the partition tables to resolve that. This is mostly a SWAG; I
> didn't trace the code to this point.

Also, there was a recent patch changed in gentoo-sources to prevent
PARTUUID from appearing in /proc/self/mountinfo. This would explain
why this is a "new" behavior for you.

https://bugs.gentoo.org/show_bug.cgi?id=467266