Re: [gentoo-dev] separate /usr without initramfs

[Jaco Kroon <jaco@uls.co.za>]

Hi,

On 2019/10/25 20:14, William Hubbs wrote:
> Hey all,
>
> I have been advised to bring this topic back to the list before taking
> any action, so here it is.
>
> First, I need to clarify what I'm *NOT* talking about.
>
> This discussion has nothing to do with whether or not you have the
> split-usr use flag turned on; all of us officially have that on because
> /bin, /lib* and /sbin are directories in the official Gentoo setup. In
> other words, I am *not* talking about forcing the /usr merge.
>
> Unfortunately, the concept of separate usr has gotten wrapped up in the
> split-usr use flag and doesn't have to be.  For the record, I mean something
> very specific when I say "separate usr". I am talking about the situation
> where /usr is a mount point separate from /, so in this thread, let's stick
> to "separate usr" for that situation. I am *not* even saying that using
> separate usr is wrong or unsupported. You can even run separate usr with
> split-usr turned off if you would like to do so.
>
> Now for the use case I want to talk about, and that is using separate
> /usr without using an initramfs to boot your system and pre-mount /usr.
>
> If you do this, many things are broken, and this is why the binary
> distros all use an initramfs if you do this. This configuration is also
> unsupported officially in Gentoo [1] [2], and it is not shown as the
> example setup in our handbook.
>
> I want to hear from people who have / and /usr on separate partitions
> and who are not using an initramfs.
>
> If you are in this group, I have a very specific question. Why aren't
> you using an initramfs?

Because until recently it wasn't an issue.  So for me the final kicker 
was still not a separate /usr.  For my use case everything except a big 
warning about md5sum not being available during boot works.  This is NOT 
desktop setup for MOST of my systems.  On the few that are I don't 
generally have things like bluetooth keyboards etc that I need available 
during early(ish) boot or anything crazy.  So the argument that "many 
things break" may be accurate, but I've yet to find a concrete example 
that bugs me enough to care.

There is basically one thing that I found that broke "out of the box" 
with a separate /usr - and that's if I have one of those stupid LTE 
modem things that pretend to be a disk drive/cdrom or something similar 
until you tell it to switch to network mode. The same thing that, for 
me, breaks with suspend resume (after resume I have to kill 
modem_manager, and start it *before* replugging the modem or it simply 
will not work - another discussion).

So frankly, I just don't see the benefit.  The reason I've eventually 
bothered with setting up and creating an initramfs now was because of 
/lib/firmware which I need available during module load (pre 
/etc/init.d/localmount) so that firmware is available as soon as amdgpu 
and i915 loads or else I end up with a borked screen.  Only uefi fb 
compiled into the kernel (required to get hand-over from grub2 ... at 
least, the only way I could get it to work smoothly).

The initramfs we ONLY pull in on systems where it's required (one 
currently, the machine I'm typing this on).

Why do I have /lib/firmware on a separate partition now?  Because 512MB 
for / used to be plenty, and I have MANY history systems out there which 
is non-trivial to make partitioning changes to (I keep / in a raw 
partition).  Now just /lib/firmware is larger than that.

Like William, / and /boot are definite partitions, and I want to keep 
these small.  Everything else is LVM.  Most systems have >50% of VG 
space unallocated because people always overestimate what they really need.

Separate partitions means I can set up separate mount options 
(nosuid,nodev etc ...)

Like William, I feel more cogs means more opportunities for breakage. I 
roll my own vanilla kernel, with one or two of my own patches.  I roll 
my own init script for initramfs.  Why - because system bootability is 
of utmost importance.  So I absolutely have to KNOW that it'll work, and 
when it doesn't that I can walk someone through fixing it over the 
phone. Never used the default gentoo initramfs.  genkernel always pulls 
in stuff I don't want nor need.  Sometimes it's just simpler I guess.  
But that's the thing - Gentoo gives me CHOICE.  Which I don't get from 
other distributions.  These are not the kind of things I like to leave 
to chance, or in the hands of a continuously changing tool (eg, dracut 
as mentioned by William).

Recently we ran into a bug causing filesystem corruption on /usr/portage 
(for some reason it was always under /usr/portage). With /usr on a 
separate partition we could recover that by setting appropriate boot 
options *remotely*.  No need to drive out.  Some of these systems were 
 >100km away.

So another motivation for separate / and /usr for me is that / is much 
more read-heavy than /usr, and as such, with the lower write ratio lower 
risk of corruption.  Better ability to recover.

As to why not use the initramfs - simple:  It's not needed.

The only potential advantage in my opinion is if you can build a 
recovery system in there that's small enough, and contains all 
conceivable tools required to recover from just about any boot failure.  
Work-in-progress I guess.

I'm not sure that answers your question, and this is one of those 
debates that can run in circles for hours, days and even weeks. So I 
hope I at least managed to give you some insight into my thinking and 
reasoning.

Kind Regards,
Jaco