[gentoo-amd64] busybox-1.23.0 not assembling RAID root

[gentoo-amd64] busybox-1.23.0 not assembling RAID root

[Mark Knecht]

Hi,
   There was an update to stable busybox this weekend choosing version
1.23.0 as the new stable version. Since this update, for NEW kernels
only, I'm unable to assemble my RAID6 root device and therefore the
kernel won't boot. Has anyone else had similar problems? I've looked
around for recent posts, etc., but haven't found anything as of yet
but the gatekeeper in the Gentoo Bugzilla environment won't let me
file a bug report.

   At issue here is assembling the root RAID device. I build the
initramfs into the kernel for exactly this sort of reason. Other than
changing a few RAID UUIDs this setup hasn't changed in years. I masked
1.23.0, went back to 1.21.0 and that still works for new kernel so it
appears to be a busybox issue.

   My configuration follows. initramfs.config is referenced in my
kernel config file which then loads initramfs_init_new.sh to do the
work. The failure is three messages:

/init: line 16: mount: not found
/init: line 17: mount: not found
/init: line 18: mount: not found

which appear to correlate with lines 16-18:

mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev

   Any input are greatly appreciated.

Thanks,
Mark



mark@c2RAID6 ~ $ cat /usr/src/initramfs.config
dir /bin 755 0 0
file /bin/busybox /bin/busybox 755 0 0
slink /bin/sh busybox 777 0 0

dir /realroot 755 0 0
dir /etc 755 0 0
dir /proc 755 0 0
dir /sys 755 0 0

dir /sbin 755 0 0
file /sbin/mdadm /sbin/mdadm 755 0 0

file /sbin/e2fsck /sbin/e2fsck 755 0 0
dir /lib 755 0 0
file /lib/libext2fs.so /usr/lib64/libext2fs.so 755 0 0
file /etc/mdadm.conf /usr/src/mdadm_initramfs.conf 755 0 0

dir /dev 755 0 0
nod /dev/console 600 0 0 c 5 1
nod /dev/null 666 0 0 c 1 3
nod /dev/tty 666 0 0 c 5 0
nod /dev/urandom 666 0 0 c 1 9

file /init /usr/src/initramfs_init_new.sh 755 0 0
mark@c2RAID6 ~ $




mark@c2RAID6 ~ $ cat /usr/src/initramfs_init_new.sh
#!/bin/busybox sh

rescue_shell() {
    echo "-----"
    echo "!!!!!"
    echo "Something went wrong."
    echo "Dropping into busybox"
    echo "!!!!!"
    echo "-----"
    busybox --install -s
    exec /bin/sh
}


# Mount the /proc and /sys filesystems.
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev

# Do your stuff here.
echo "   "
echo "This init script assembles the RAID6 root as /dev/md3"
echo "mounts it as rootfs and, if there are no problems,"
echo "boots it up, else calls rescue_shell/busybox"
echo "   "

mdadm --assemble /dev/md3

# Mount the root filesystem.
mount -o ro /dev/md3 /realroot  || rescue_shell

# Clean up.
umount /dev
umount /proc
umount /sys

# Boot the real thing.
exec switch_root /realroot /sbin/init
mark@c2RAID6 ~ $

Re: [gentoo-amd64] busybox-1.23.0 not assembling RAID root

[bitlord]

On Mon, 19 Jan 2015 13:55:30 -0800
Mark Knecht <markknecht@gmail.com> wrote:

> Hi,
>    There was an update to stable busybox this weekend choosing version
> 1.23.0 as the new stable version. Since this update, for NEW kernels
> only, I'm unable to assemble my RAID6 root device and therefore the
> kernel won't boot. Has anyone else had similar problems? I've looked
> around for recent posts, etc., but haven't found anything as of yet
> but the gatekeeper in the Gentoo Bugzilla environment won't let me
> file a bug report.
> 
>    At issue here is assembling the root RAID device. I build the
> initramfs into the kernel for exactly this sort of reason. Other than
> changing a few RAID UUIDs this setup hasn't changed in years. I masked
> 1.23.0, went back to 1.21.0 and that still works for new kernel so it
> appears to be a busybox issue.
> 
>    My configuration follows. initramfs.config is referenced in my
> kernel config file which then loads initramfs_init_new.sh to do the
> work. The failure is three messages:
> 
> /init: line 16: mount: not found
> /init: line 17: mount: not found
> /init: line 18: mount: not found
> 

No RAID here (LUKS(LVM())), but I have custom initramfs, and I see this
problem after update + regenerating initramfs (with new binaries), which
in the end produce a kernel panic ... (I don't know much))
Older images work fine, I did some "testing" (guessing), and adding
before all those mount ... lines  
'/bin/busybox --install -s'
(Which should "install" all symlinks for supported tools?)
That generates a lot of other messages but I was able to boot my
system with new image.

Re: [gentoo-amd64] busybox-1.23.0 not assembling RAID root

[Mark Knecht]

On Mon, Jan 19, 2015 at 6:13 PM, bitlord <bitlord0xff@gmail.com> wrote:
> On Mon, 19 Jan 2015 13:55:30 -0800
> Mark Knecht <markknecht@gmail.com> wrote:
>
>> Hi,
>>    There was an update to stable busybox this weekend choosing version
>> 1.23.0 as the new stable version. Since this update, for NEW kernels
>> only, I'm unable to assemble my RAID6 root device and therefore the
>> kernel won't boot. Has anyone else had similar problems? I've looked
>> around for recent posts, etc., but haven't found anything as of yet
>> but the gatekeeper in the Gentoo Bugzilla environment won't let me
>> file a bug report.
>>
>>    At issue here is assembling the root RAID device. I build the
>> initramfs into the kernel for exactly this sort of reason. Other than
>> changing a few RAID UUIDs this setup hasn't changed in years. I masked
>> 1.23.0, went back to 1.21.0 and that still works for new kernel so it
>> appears to be a busybox issue.
>>
>>    My configuration follows. initramfs.config is referenced in my
>> kernel config file which then loads initramfs_init_new.sh to do the
>> work. The failure is three messages:
>>
>> /init: line 16: mount: not found
>> /init: line 17: mount: not found
>> /init: line 18: mount: not found
>>
>
> No RAID here (LUKS(LVM())), but I have custom initramfs, and I see this
> problem after update + regenerating initramfs (with new binaries), which
> in the end produce a kernel panic ... (I don't know much))
> Older images work fine, I did some "testing" (guessing), and adding
> before all those mount ... lines
> '/bin/busybox --install -s'
> (Which should "install" all symlinks for supported tools?)
> That generates a lot of other messages but I was able to boot my
> system with new image.
>

Hi bitlord,
   Thanks very much for the ideas. I suspect it's going to get me
closer when I can work on this again this evening. I'm going quite
slow as it would be really bad if something I do in this area ends up
hurting the machine.

   One idea I had after reading your response was instead of doing the
complete '--install -s' maybe I'd add just one line

slink /bin/mount busybox 755 0 0

in the main setup file to get the mount command and see if that solves
the problem. However your solution is likely better as when I fall
into busybox I suspect I'd have everything busybox provides which
would be more powerful.

   Maybe you can add a response to my bug report here?

https://bugs.gentoo.org/show_bug.cgi?id=536988

The Jeroen guy who acts as gatekeeper is marking the bug report as
invalid because no one else is reporting anything. I think that's
pretty bogus so anything you can add would likely be helpful.

   None the less, thanks for the response. It's made me revisit how
this works (at the 10000 foot level) which is good.

Cheers,
Mark

Re: [gentoo-amd64] busybox-1.23.0 not assembling RAID root

[Douglas J Hunley]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

On Tue, Jan 20, 2015 at 5:43 PM, Mark Knecht <markknecht@gmail.com> wrote:

>    Maybe you can add a response to my bug report here?
>
> https://bugs.gentoo.org/show_bug.cgi?id=536988
>

I have the same issue and my comment would look almost exactly like
bitlord's comment on that bug. To fix it, I changed:
mount -t proc none /proc
to:
busybox mount -t proc none /proc

and the problem went away. However, I don't think that should be needed as
the script is run by #!/bin/busybox


-- 
Douglas J Hunley (doug.hunley@gmail.com)
Twitter: @hunleyd                                               Web:
about.me/douglas_hunley
G+: http://google.com/+DouglasHunley

[-- Attachment #2: Type: text/html, Size: 1563 bytes --]

Re: [gentoo-amd64] busybox-1.23.0 not assembling RAID root

[Mark Knecht]

On Wed, Jan 21, 2015 at 9:28 AM, Douglas J Hunley <doug.hunley@gmail.com> wrote:
>
> On Tue, Jan 20, 2015 at 5:43 PM, Mark Knecht <markknecht@gmail.com> wrote:
>>
>>    Maybe you can add a response to my bug report here?
>>
>> https://bugs.gentoo.org/show_bug.cgi?id=536988
>
>
> I have the same issue and my comment would look almost exactly like
> bitlord's comment on that bug. To fix it, I changed:
> mount -t proc none /proc
> to:
> busybox mount -t proc none /proc
>
> and the problem went away. However, I don't think that should be needed as
> the script is run by #!/bin/busybox
>

For those not following the bug report there was a change introduced
by the busybox package maintainer wherein the 1.21 ebuild changed from
allyesconfig to defconfig. I never used 1.21 so I didn't see it until
we got a stable upgrade.

The change I made was adding

/bin/busybox --install -s

immediately before my mount commands. Please note that without the
explicit /bin/busybox the busybox binary wasn't found and nothing was
fixed.

Thanks for your help.

Cheers,
Mark

Re: [gentoo-amd64] busybox-1.23.0 not assembling RAID root

[Randy Barlow]

On January 19, 2015 4:55:30 PM EST, Mark Knecht <markknecht@gmail.com> wrote:
>The failure is three messages:
>
>/init: line 16: mount: not found
>/init: line 17: mount: not found
>/init: line 18: mount: not found
>
>which appear to correlate with lines 16-18:
>
>mount -t proc none /proc
>mount -t sysfs none /sys
>mount -t devtmpfs none /dev

Have you tried using a full path to the mount command?

[gentoo-amd64] Re: busybox-1.23.0 not assembling RAID root

[Duncan]

Randy Barlow posted on Mon, 19 Jan 2015 21:34:54 -0500 as excerpted:

> On January 19, 2015 4:55:30 PM EST, Mark Knecht <markknecht@gmail.com>
> wrote:
>>The failure is three messages:
>>
>>/init: line 16: mount: not found
>>/init: line 17: mount: not found
>>/init: line 18: mount: not found
>>
>>which appear to correlate with lines 16-18:
>>
>>mount -t proc none /proc
>>mount -t sysfs none /sys
>>mount -t devtmpfs none /dev
> 
> Have you tried using a full path to the mount command?

Disclaimer:  I've not had busybox on my system since I switched to gentoo 
in 2004[1], so don't expect me to be much help with busybox-specific 
stuff.

That said, the error is clear enough...  mount isn't found on the path.  
That's almost certainly one of two problems, both with straightforward 
solutions:

1) Missing mount -> busybox symlink.  (Most likely.)

2) That symlink not in the path, for some reason.


If the symlink's in the initr*, that'd be #2, so either adjust the path 
accordingly before calling mount, or as Randy suggests, call mount using 
the full path.

If instead the symlink's not there at all, as I suspect, that's #1. 
Ensure that the symlink is available before attempting to use it, either 
by placing it in the initr* at initr* build-time, or by using bitlord's 
suggestion (busybox --install -s prior to invoking mount).

Presumably something changed in the new busybox and it's documented 
somewhere, for example that initr*s are expected to run busybox --install 
-s if they use the mount symlink now, or the like, and if I used busybox 
I'd probably know about it from some time ago since I run ~arch, and 
could then explain what and why, but since I don't... well I already said 
"presumably".

---
[1] Busybox wouldn't build then for some reason so I bypassed/
package.provided it, and I've never needed it as I keep what amounts to a 
second copy of the system as a backup, with the copy taken when the 
system was known working and stable, and tested by booting with root= 
pointing at the backup before I trust it just as I would in an emergency, 
as my emergency-boot, so I've simply never bothered with busybox.  Of 
course these days I run a custom profile with a totally empty @system 
(all normal @system packages have a negating entry leaving nothing in 
@system), and all the not-otherwise-depended packages I actually need in 
@world, and since I don't need busybox and it's not a dependency of 
anything I run, it's simply not there. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-amd64] busybox-1.23.0 not assembling RAID root

[Mark Knecht]

On Mon, Jan 19, 2015 at 6:34 PM, Randy Barlow
<randy@electronsweatshop.com> wrote:
> On January 19, 2015 4:55:30 PM EST, Mark Knecht <markknecht@gmail.com> wrote:
>>The failure is three messages:
>>
>>/init: line 16: mount: not found
>>/init: line 17: mount: not found
>>/init: line 18: mount: not found
>>
>>which appear to correlate with lines 16-18:
>>
>>mount -t proc none /proc
>>mount -t sysfs none /sys
>>mount -t devtmpfs none /dev
>
> Have you tried using a full path to the mount command?
>
>

I have not. When I rebuild with 1.23 I'll try making no changes,
booting to the point where I drop into busybox and then I'll look
around at whether mount is even in there.

Thanks!

Cheers,
Mark