[gentoo-user] udev detection weirdness

[gentoo-user] udev detection weirdness

[Daniel Frey]

Some time ago (more than a year ago) one of my computers failed to boot
after an update (note - not a kernel update. A @world update.) I
couldn't figure out why and installed grub2 and used UUIDs as a
temporary fix.

I got a new SSD for that machine today, so I went and moved everything
over, then (of course) it wouldn't boot any longer, and I remembered my
kludge and spent some time finally figuring out the cause.

It appears to be udev. Somewhere along in its stupid detection it
decides to process USB devices before sata ports, thusly randomly
renaming the boot drive to something else in the process.

It took me forever to figure this out, I eventually had a lightbulb
moment and used my phone to record video of it booting, then slowing it
down, as when the kernel panics you can't scroll back up to see WTF
happened.

This is an older machine, but I'm not convinced it's the motherboard
doing this. I've checked the boot order in the BIOS. I've also tried
setting and unsetting "BIOS order determines boot disk" in the kernel
config and it made no difference.

What eventually fixed it was building USB as modules. (Another kludge!)

I have no custom udev rules, the only rules I could find were in
/lib/udev/rules.d:

10-dm.rules
11-dm-lvm.rules
13-dm-disk.rules
40-gentoo.rules
50-udev-default.rules
60-block.rules
60-cdrom_id.rules
60-drm.rules
60-evdev.rules
60-persistent-alsa.rules
60-persistent-input.rules
60-persistent-storage-tape.rules
60-persistent-storage.rules
60-persistent-v4l.rules
60-serial.rules
64-btrfs.rules
69-dm-lvm-metad.rules
70-infrared.rules
70-mouse.rules
70-udev-acl.rules
75-net-description.rules
75-probe_mtd.rules
78-sound-card.rules
80-drivers.rules
80-net-setup-link.rules
80-udisks.rules
90-alsa-restore.rules
90-network.rules
95-dm-notify.rules
99-nvidia.rules

Does anyone have any explanation for this daft behaviour or know where I
should look?

I have multiple machines and it's only this one that has this problem,
which happened after a @world update long ago.

Dan

[gentoo-user] Re: udev detection weirdness

[James]

Daniel Frey <djqfrey <at> gmail.com> writes:


> It appears to be udev. Somewhere along in its stupid detection it
> decides to process USB devices before sata ports, thusly randomly
> renaming the boot drive to something else in the process.

> It took me forever to figure this out, I eventually had a lightbulb
> moment and used my phone to record video of it booting, then slowing it
> down, as when the kernel panics you can't scroll back up to see WTF
> happened.

Kernel crash dumps might help [1]



> This is an older machine, but I'm not convinced it's the motherboard
> doing this. I've checked the boot order in the BIOS. I've also tried
> setting and unsetting "BIOS order determines boot disk" in the kernel
> config and it made no difference.

You might want to 'emerge -1 sys-apps/hwids'


> What eventually fixed it was building USB as modules. (Another kludge!)

There are numerous 'usb sniffers' that  capture data. Some clue
might be found using a usb sniffer.



> I have no custom udev rules, the only rules I could find were in
> /lib/udev/rules.d:

I use sys-fs/eudev.   ymmv.


> Does anyone have any explanation for this daft behaviour or know where I
> should look?
> I have multiple machines and it's only this one that has this problem,
> which happened after a  <at> world update long ago.

If you have a similar setup on similar hardware, then 'diff' the  (dmesg)
boot log files for any differences and analyze.


hth,
James

[1] https://wiki.gentoo.org/wiki/Kernel_Crash_Dumps

[gentoo-user] Re: udev detection weirdness

[Hans]

On 26/05/16 23:39, James wrote:
> Daniel Frey <djqfrey <at> gmail.com> writes:
>
>
>> It appears to be udev. Somewhere along in its stupid detection it
>> decides to process USB devices before sata ports, thusly randomly
>> renaming the boot drive to something else in the process.
>
>> It took me forever to figure this out, I eventually had a lightbulb
>> moment and used my phone to record video of it booting, then slowing it
>> down, as when the kernel panics you can't scroll back up to see WTF
>> happened.
>
> Kernel crash dumps might help [1]
>
>
>
>> This is an older machine, but I'm not convinced it's the motherboard
>> doing this. I've checked the boot order in the BIOS. I've also tried
>> setting and unsetting "BIOS order determines boot disk" in the kernel
>> config and it made no difference.
>
> You might want to 'emerge -1 sys-apps/hwids'
>
>
>> What eventually fixed it was building USB as modules. (Another kludge!)
>
> There are numerous 'usb sniffers' that  capture data. Some clue
> might be found using a usb sniffer.
>
>
>
>> I have no custom udev rules, the only rules I could find were in
>> /lib/udev/rules.d:
>
> I use sys-fs/eudev.   ymmv.
>
>
>> Does anyone have any explanation for this daft behaviour or know where I
>> should look?
>> I have multiple machines and it's only this one that has this problem,
>> which happened after a  <at> world update long ago.
>
> If you have a similar setup on similar hardware, then 'diff' the  (dmesg)
> boot log files for any differences and analyze.
>
>
> hth,
> James
>
> [1] https://wiki.gentoo.org/wiki/Kernel_Crash_Dumps
>
>
>
I had similar problems. Fixed them permanently by using disk labels on 
all partitions on all computers.

Re: [gentoo-user] udev detection weirdness

[Stroller]

> On Thu, 26 May 2016, at 7:14 am, Daniel Frey <djqfrey@gmail.com> wrote:
> 
> … I couldn't figure out why and installed grub2 and used UUIDs as a
> temporary fix.
> 
> … It appears to be udev. Somewhere along in its stupid detection it
> decides to process USB devices before sata ports, thusly randomly
> renaming the boot drive to something else in the process.

I don't understand.

Surely this is why it's correct to use UUIDs or volume labels?

A person shouldn't care what the kernel calls his hard-drives, nor in which order it enumerates its USB devices.


> What eventually fixed it was building USB as modules. (Another kludge!)

I don't understand why this is a kludge. I build everything I can as modules, so that no unneeded drivers are loaded.

Stroller.