[gentoo-user] How to set up drive with many Linux distros?

[gentoo-user] How to set up drive with many Linux distros?

[Grant Edwards]

For many years, I've used a hard drive on which I have 8-10 Linux
distros installed -- each in a separate (single) partition.

There is also a single swap partition (used by all of the different
Linux installations).

There is also a small partition devoted only to the "master" instance
of Grub that lives in the MBR and the space between the MBR and the
first partition (the drive uses a DOS disklabel). That master instance
of Grub has a menu which contains entries which "chainload" each of
the other partitions.

For many years, this worked great. All of the various distro
installers offered the option of installing the bootloader in the MBR
(e.g. /dev/sda) or in a partition (e.g /dev/sdaN).  I would tell the
installer to install the bootloader in the root partition, and
everything "just worked".

However, the choice to install bootloaders in partitions instead of
the MBR has been removed from most (all?) of the common installers.
This forces me to jump through hoops when installing a new Linux
distro:

 1. Back up the MBR and gap between the MBR and the first partition.

 2. Let the installer install it's bootloader (seems it's always grub
    these days) in the MBR.

 3. Boot into the newly installed Linux.

 4. Manually install grub in the root partition (e.g. /dev/sdaN) using
    the --force option to tell grub to use blocklists to find it's
    files.

 5. Find those grub files and lock them so they can't be moved.

 6. Restore the MBR/gap backup from step 1.

It seems like there should be a better way to do this.  One might hope
that UEFI offers a solution to this problem. Google has found me
others asking the same question but no real answers.

Is there an easier way to do this?

--
Grant

Re: [gentoo-user] How to set up drive with many Linux distros?

[Wol]

On 22/02/2024 19:17, Grant Edwards wrote:
> However, the choice to install bootloaders in partitions instead of
> the MBR has been removed from most (all?) of the common installers.
> This forces me to jump through hoops when installing a new Linux
> distro:

File a bug!

If that's true, it basically borks any sort of dual boot, unusual disk 
layout, whatever.

Last time I installed SUSE, it trashed my boot totally because it didn't 
recognise my disk stack, failed to load necessary drivers, and worse 
trashed my gentoo boot too ...

Cue one big rescue job to get the system up and working again. At least 
it was only the boot that was trashed.

Cheers,
Wol

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-22, Wol <antlists@youngman.org.uk> wrote:
> On 22/02/2024 19:17, Grant Edwards wrote:
>
>> However, the choice to install bootloaders in partitions instead of
>> the MBR has been removed from most (all?) of the common installers.
>> This forces me to jump through hoops when installing a new Linux
>> distro:
>
> File a bug!

LOL, good one! As if a normal person filing a bug with RedHat or
Ubuntu actually accomplishes anything.  I'll tell them to make systemd
optional while I'm at it.

> If that's true, it basically borks any sort of dual boot, unusual disk 
> layout, whatever.

Yep, it does. The answer from <distro> is:

   You should really just (shut up and) install <distro>'s (obviously
   superior) bootloader in the MBR.  It will auto-detect (some of) the
   other already installed (obviously inferior) OSes, and will add
   (some subset of) them to the boot menu that will (sometimes) allow
   you to boot them (maybe -- if you kneel, bow your head and ask
   nicely).

> Last time I installed SUSE, it trashed my boot totally because it
> didn't recognise my disk stack, failed to load necessary drivers,
> and worse trashed my gentoo boot too...
>
> Cue one big rescue job to get the system up and working again. At
> least it was only the boot that was trashed.

I've been reading up on UEFI, and it doesn't seem to be any
better. People complain about distro's stomping on each other's files
in the ESP partiton and multiple distro's using the same name in the
boot slots stored in NVM. And then the boot choice order changes
(though it may not be apparent to the naked eye) when one of the
distros decides to update/reinstall its boot stuff.

--
Grant

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Wol]

On 22/02/2024 21:45, Grant Edwards wrote:
> I've been reading up on UEFI, and it doesn't seem to be any
> better. People complain about distro's stomping on each other's files
> in the ESP partiton and multiple distro's using the same name in the
> boot slots stored in NVM. And then the boot choice order changes
> (though it may not be apparent to the naked eye) when one of the
> distros decides to update/reinstall its boot stuff.

At least if you use UEFI *as* your bootloader, then that won't happen. 
That assumes you're using UEFI, though!

In which case, <distro>'s bootloader doesn't get a look-in.

As for "<distro>'s obviously superior bootloader", well <other distro> 
is using the exact same boot-loader, and when IT installs, how is it 
going to be able to boot <distro> if it can't call <distro>'s boot 
loader because it's just trashed it by overwriting it?

To me, you seem to be describing the *default* installer setup, that's 
been there for ever. Last I installed SUSE, iirc I had to specify 
"advanced bootloader installation", most of who's options I didn't even 
understand!, but it did do what I told it to (apart from not recognising 
my weird disk stack!).

If you can find, and understand!, that advanced options, I think you'll 
find you can do what you want.

Cheers,
Wol

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-22, Wol <antlists@youngman.org.uk> wrote:
> On 22/02/2024 21:45, Grant Edwards wrote:
>> I've been reading up on UEFI, and it doesn't seem to be any
>> better. People complain about distro's stomping on each other's files
>> in the ESP partiton and multiple distro's using the same name in the
>> boot slots stored in NVM. And then the boot choice order changes
>> (though it may not be apparent to the naked eye) when one of the
>> distros decides to update/reinstall its boot stuff.
>
> At least if you use UEFI *as* your bootloader, then that won't
> happen.  That assumes you're using UEFI, though!

According to what I've read UEFI isn't a bootloader. It's a boot
manager which can load and run EFI bootloaders (of which there can be
multiple installed).

> In which case, <distro>'s bootloader doesn't get a look-in.

Yes, AFAICT, it does (sometimes?). When you install <distro> under
UEFI it installs EFI bootloader files (either kernels wrapped in EFI
bootloader executables or the grub EFI bootloader) in the EFI Systgem
Partition (ESP), and then adds one or more entries in the EFI NVM that
points to those files (or something like that).  The Linux UEFI
systems I have all still use grub2 (which gets written to the ESP).

It's entirely possible for one distro to overwrite files in the ESP
that belong to other distros. I've read multiple complaints about
exactly that when trying to do multi-boot with UEFI. In practice it's
just like the fight over who owns the MBR and the DOS disklable gap.

One recipe I read about for doing what I wanted to do with UEFI
involved installing a Linux distro (didn't really matter which), then
installing rEFInd. After that, some manual renaming and deleting of
the files in the ESP was required. Then he started installing various
distros. After each distro installation, the author had to re-install
rEFInd, and after many of them he had to manually remove or rename
files in the ESP (or adjust the rEFInd config file).

And in the end, he ended up with multiple menu entries (for different
installations) that had identical names.

It was more complicated and difficult than my current scheme.

> As for "<distro>'s obviously superior bootloader", well <other distro> 
> is using the exact same boot-loader, and when IT installs, how is it 
> going to be able to boot <distro> if it can't call <distro>'s boot 
> loader because it's just trashed it by overwriting it?

In my experience, <distro>'s bootloader does not boot other
installations by calling other bootloaders. It does so by rummaging
through all of the other partitions looking for kernel images, intird
files, grub.cfg files, etc.  It then adds menu entries to the config
file for <distro>'s bootloader which, when selected, directly load the
kernel image and initrd from those other partitions. Sometimes, it
works -- at least until those other installations get updated without
the knowlege of the distro that currently "owns" the MBR's bootloader
config. Then it stops working until you tell that bootloader to re-do
it's rummaging about routine.

> To me, you seem to be describing the *default* installer setup, that's 
> been there for ever. Last I installed SUSE, iirc I had to specify 
> "advanced bootloader installation", most of who's options I didn't even 
> understand!, but it did do what I told it to (apart from not recognising 
> my weird disk stack!).

So SuSe still allows you to install grub to a partition instead of
MBR? That's encouraging. RH and Ubuntu used to allow that, but AFAIK,
now they do not.

> If you can find, and understand!, that advanced options, I think
> you'll find you can do what you want.

I'd welcome pointers to where those advanced options are in the RH and
Ubunutu installers -- I've searched everywhere I can think of. Various
things Google has found lead me to believe that they no longer support
installing grub in a partition.

I guess I'll stick with my current setup.

Or perhaps I'll switch from a DOS disklabel to a GPT disklabel.
Instead of backing up and restoring the MBR and the gap, I would
backup and restore the MBR and the BIOS boot partition. And I could
use UUIDs and partition labels.

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Wojciech Kuzyszyn]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

Hello!

I guess most (all) of the distro's you are talking about use GRUB (or
at least they allow to do it). If that's true, I'm pretty sure you can
happily let them overwrite the GRUB in MBR as many times as they want,
since it's the same (or just probably minor version differences)
bootloader. Just make a copy of /boot/grub/grub.cfg and make sure it's
the same on every partition. Or, even better, if that's possible right
now, make a common /boot partition and after installing the new distro
just merge the (probably new) /boot/grub/grub.cfg with your old one.

I really think that *should* work!

Take care,
ks. Wojciech Kuzyszyn

[-- Attachment #2: Podpis cyfrowy OpenPGP --]
[-- Type: application/pgp-signature, Size: 277 bytes --]

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-23, Wojciech Kuzyszyn <wkuz@op.pl> wrote:

> I guess most (all) of the distro's you are talking about use GRUB (or
> at least they allow to do it).

Yes, I belive that they are all now using Grub2.

> If that's true, I'm pretty sure you can happily let them overwrite
> the GRUB in MBR as many times as they want, since it's the same (or
> just probably minor version differences) bootloader. Just make a
> copy of /boot/grub/grub.cfg and make sure it's the same on every
> partition.

That means I have to update all of those grub.cfg files manually every
time I install or update a distro. That's a lot of work.

> Or, even better, if that's possible right now, make a common /boot
> partition and after installing the new distro just merge the
> (probably new) /boot/grub/grub.cfg with your old one.

"just merge the new grub.cfg file with the old one" is the problem.

I tried that for a while: every time a distro got installed, I would
add it to the "main" grub config file. Every time a distro got a
kernel update, I'd modify the main grub config file. It was a lot more
work that my current scheme (and a lot more error prone).

> I really think that *should* work!

It would work, but maintaining the grub.cfg files is a lot of
work. The scheme I'm using now doens't require me to mess with any of
the grub.cfg files when distros get updated or installed.

--
Grant

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Michael]

[-- Attachment #1: Type: text/plain, Size: 7519 bytes --]

On Friday, 23 February 2024 00:28:59 GMT Grant Edwards wrote:
> On 2024-02-22, Wol <antlists@youngman.org.uk> wrote:
> > On 22/02/2024 21:45, Grant Edwards wrote:
> >> I've been reading up on UEFI, and it doesn't seem to be any
> >> better. People complain about distro's stomping on each other's files
> >> in the ESP partiton and multiple distro's using the same name in the
> >> boot slots stored in NVM. And then the boot choice order changes
> >> (though it may not be apparent to the naked eye) when one of the
> >> distros decides to update/reinstall its boot stuff.
> > 
> > At least if you use UEFI *as* your bootloader, then that won't
> > happen.  That assumes you're using UEFI, though!
> 
> According to what I've read UEFI isn't a bootloader. It's a boot
> manager which can load and run EFI bootloaders (of which there can be
> multiple installed).

The UEFI firmware of the MoBo contains its own bootloader and a boot manager 
with its own boot menu, initialised and running from the MoBo's EEPROM. Unlike 
conventional/legacy bootloaders stored in the first sector of a disk (MBR), 
the UEFI firmware is stored in the MoBo's EEPROM, a larger equivalent to the 
old CMOS.

However, this comes with the caveat the UEFI 'bootloader' can only load .efi 
executables which have already been placed in the FAT32 formatted EFI System 
Partition (ESP).  Unlike GRUB's MBR Stage 1 bootloader code, the UEFI firmware 
is large enough to contain its own fs driver to be able to read the ESP fs 
content and identify and run all .efi applications.

Kernel images which have been built with the EFI stub and therefore masquerade 
as .efi compatible files can be loaded directly by the UEFI firmware without 
the need of an additional bootloader.


> > In which case, <distro>'s bootloader doesn't get a look-in.
> 
> Yes, AFAICT, it does (sometimes?). When you install <distro> under
> UEFI it installs EFI bootloader files (either kernels wrapped in EFI
> bootloader executables or the grub EFI bootloader) in the EFI Systgem
> Partition (ESP), and then adds one or more entries in the EFI NVM that
> points to those files (or something like that).  The Linux UEFI
> systems I have all still use grub2 (which gets written to the ESP).

Legacy GRUB on an MBR partitioned disk, writes its Stage 1 bootloader code in 
the first sector and Stage 1.5 with the fs drivers in sector 34, before the 
first partition.

GRUB2 on a UEFI system installs the file /efi/<distro>/grubx64.efi in the ESP, 
an equivalent of the Stage 1 and Stage 1.5 of legacy GRUB.  The Stage 2 /boot/
grub/ files can be installed either in the ESP, or on a different partition.


> It's entirely possible for one distro to overwrite files in the ESP
> that belong to other distros. I've read multiple complaints about
> exactly that when trying to do multi-boot with UEFI. In practice it's
> just like the fight over who owns the MBR and the DOS disklable gap.

It doesn't really matter if the grubx64.efi executable is overwritten, as long 
as the OS_PROBER is not disabled in /etc/default/grub.  Re-running grub-
mkconfig will re-scan the ESP and any drive/partitions, pick up any OS kernel 
images known by GRUB and add them to GRUB's boot menu.

The problem starts if/when kernel images are overwritten by successive Linux 
OS distros.  This is likely when derivatives of the same main distros e.g. 
Ubuntu all create a directory called /EFI/ubuntu/ in the ESP and drop their 
kernels & initrd images in there potentially overwriting other distro's files.


> One recipe I read about for doing what I wanted to do with UEFI
> involved installing a Linux distro (didn't really matter which), then
> installing rEFInd. After that, some manual renaming and deleting of
> the files in the ESP was required. Then he started installing various
> distros. After each distro installation, the author had to re-install
> rEFInd, and after many of them he had to manually remove or rename
> files in the ESP (or adjust the rEFInd config file).
> 
> And in the end, he ended up with multiple menu entries (for different
> installations) that had identical names.

The concept of one bootloader/manager ruling them all is broadly the same 
whether you use rEFInd or GRUB, as are the hoops you have to jump through to 
accommodate distros' automated/hardcoded installation scripts.

When using a distro's installer menu on a legacy BIOS MoBo you can select a 
partition (PBR) to install GRUB, but GRUB will complain and suggest you could 
use blocklists but it is unreliable.  Last time I received an error like this, 
I installed grub in a PBR manually with the '--force' option, without using 
the installer GUI.  After that, whenever I updated GRUB it complained again 
about blocklists, but it worked fine.


> It was more complicated and difficult than my current scheme.
> 
> > As for "<distro>'s obviously superior bootloader", well <other distro>
> > is using the exact same boot-loader, and when IT installs, how is it
> > going to be able to boot <distro> if it can't call <distro>'s boot
> > loader because it's just trashed it by overwriting it?
> 
> In my experience, <distro>'s bootloader does not boot other
> installations by calling other bootloaders. It does so by rummaging
> through all of the other partitions looking for kernel images, intird
> files, grub.cfg files, etc.  It then adds menu entries to the config
> file for <distro>'s bootloader which, when selected, directly load the
> kernel image and initrd from those other partitions. Sometimes, it
> works -- at least until those other installations get updated without
> the knowlege of the distro that currently "owns" the MBR's bootloader
> config. Then it stops working until you tell that bootloader to re-do
> it's rummaging about routine.
> 
> > To me, you seem to be describing the *default* installer setup, that's
> > been there for ever. Last I installed SUSE, iirc I had to specify
> > "advanced bootloader installation", most of who's options I didn't even
> > understand!, but it did do what I told it to (apart from not recognising
> > my weird disk stack!).
> 
> So SuSe still allows you to install grub to a partition instead of
> MBR? That's encouraging. RH and Ubuntu used to allow that, but AFAIK,
> now they do not.
> 
> > If you can find, and understand!, that advanced options, I think
> > you'll find you can do what you want.
> 
> I'd welcome pointers to where those advanced options are in the RH and
> Ubunutu installers -- I've searched everywhere I can think of. Various
> things Google has found lead me to believe that they no longer support
> installing grub in a partition.

Try using '--force' to make GRUB install its image in some distro's boot/root 
partition PBR instead of the disk MBR, but you'll probably have to perform 
this outside the installer script.  I've done this with VMs.


> I guess I'll stick with my current setup.
> 
> Or perhaps I'll switch from a DOS disklabel to a GPT disklabel.
> Instead of backing up and restoring the MBR and the gap, I would
> backup and restore the MBR and the BIOS boot partition. And I could
> use UUIDs and partition labels.

These days I use disks with GPT even on MoBos with legacy BIOS.  Instead of 
backing up and restoring the MBR/BIOS Boot Partition you could just reinstall 
grub and run grub-mkconfig, as long as the latter involves fewer key-presses.  
;-)

It goes without saying a backup before messing up with GRUB will save the day 
in case files are inadvertently overwritten.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-23, Michael <confabulate@kintzios.com> wrote:

> The problem starts if/when kernel images are overwritten by
> successive Linux OS distros.  This is likely when derivatives of the
> same main distros e.g.  Ubuntu all create a directory called
> /EFI/ubuntu/ in the ESP and drop their kernels & initrd images in
> there potentially overwriting other distro's files.

Yes, that's the problem that I've read about when trying to multi-boot
with UEFI.  I usually have 3-4 different Ubuntu installations and 3-4
different RedHat installations. Ubuntu in particular causes a log of
complaints about overwriting ESP files belonging to other Ubuntus.

> When using a distro's installer menu on a legacy BIOS MoBo you can
> select a partition (PBR) to install GRUB,

You used to be able to.  I can no longer find the option to do that in
Ubuntu or RedHat. I've been told that Suse still allows it.

> but GRUB will complain and suggest you could use blocklists but it
> is unreliable.  Last time I received an error like this, I installed
> grub in a PBR manually with the '--force' option, without using the
> installer GUI.  After that, whenever I updated GRUB it complained
> again about blocklists, but it worked fine.

Using --force will work fine as long as the grub 1.5 files don't get
moved afterwards. That's why I lock them in place. Locking them will
cause future upgrades to Grub for that distro to fail, but that
doens't happen very often.  When it does, you unlock them, updated
grub, re-install using --force, and lock them again.

>> I'd welcome pointers to where those advanced options are in the RH
>> and Ubunutu installers -- I've searched everywhere I can think
>> of. Various things Google has found lead me to believe that they no
>> longer support installing grub in a partition.
>
> Try using '--force' to make GRUB install its image in some distro's
> boot/root partition PBR instead of the disk MBR, but you'll probably
> have to perform this outside the installer script.  I've done this
> with VMs.

Yes, in my OP describing what I'm doing now it explains that's what I
do. Then I lock the Grub files that are located using the blocklists
created by the --force option.

>> I guess I'll stick with my current setup.
>> 
>> Or perhaps I'll switch from a DOS disklabel to a GPT disklabel.
>> Instead of backing up and restoring the MBR and the gap, I would
>> backup and restore the MBR and the BIOS boot partition. And I could
>> use UUIDs and partition labels.
>
> These days I use disks with GPT even on MoBos with legacy BIOS.

Same here -- except for this one machine.  I think I'll switch it over
soon.

> Instead of backing up and restoring the MBR/BIOS Boot Partition you
> could just reinstall grub and run grub-mkconfig, as long as the
> latter involves fewer key-presses.  ;-)

I don't use grub-mkconfig for the "main" grub. It has a fixed grub.cfg
file that does nothing but chainload the user-selected partition.

Currently, backing up MBR+gap only happens once when I install/setup
the main grub.  Restoring BMR+gap is one command (which is actually in
a shell script) that's run after any new distro is installed.

MBR+Bios-boot-partition would work pretty much the same way.

--
Grant

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Wols Lists]

On 23/02/2024 00:28, Grant Edwards wrote:
> In my experience, <distro>'s bootloader does not boot other
> installations by calling other bootloaders. It does so by rummaging
> through all of the other partitions looking for kernel images, intird
> files, grub.cfg files, etc.  It then adds menu entries to the config
> file for <distro>'s bootloader which, when selected, directly load the
> kernel image and initrd from those other partitions. Sometimes, it
> works -- at least until those other installations get updated without
> the knowlege of the distro that currently "owns" the MBR's bootloader
> config. Then it stops working until you tell that bootloader to re-do
> it's rummaging about routine.

IME distros that try that (SUSE, anyone!) generally get confused as to 
which kernel belongs to which root partition.

Hence needing to boot with a live distro to edit the resulting mess and 
get the system to actually come up without crashing ...

Cheers,
Wol

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-23, Wols Lists <antlists@youngman.org.uk> wrote:
> On 23/02/2024 00:28, Grant Edwards wrote:

>> In my experience, <distro>'s bootloader does not boot other
>> installations by calling other bootloaders. It does so by rummaging
>> through all of the other partitions looking for kernel images,
>> intird files, grub.cfg files, etc.  It then adds menu entries to
>> the config file for <distro>'s bootloader which, when selected,
>> directly load the kernel image and initrd from those other
>> partitions. Sometimes, it works -- at least until those other
>> installations get updated without the knowlege of the distro that
>> currently "owns" the MBR's bootloader config. Then it stops working
>> until you tell that bootloader to re-do it's rummaging about
>> routine.
>
> IME distros that try that (SUSE, anyone!) generally get confused as
> to which kernel belongs to which root partition.
>
> Hence needing to boot with a live distro to edit the resulting mess
> and get the system to actually come up without crashing ...

IIRC, all of the big distros used to do that. It didn't work very
well, but at least it took a really long time.

However, I read recently that Ubuntu had disabled the os-prober by
default in 22.04.  Disabling it was always one of the first things I
did after installing a new distro.

The simple solution is to give up on multi-booting a dozen different
distros on a single disk and buy a pocketful of USB 3 thumb drives.

--
Grant

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 541 bytes --]

On Fri, Feb 23, 2024 at 11:59 AM Grant Edwards <grant.b.edwards@gmail.com>
wrote:
>
>
> The simple solution is to give up on multi-booting a dozen different
> distros on a single disk and buy a pocketful of USB 3 thumb drives.
>

Given performance does drop a bit and there can be issues with allocating
hardware, why not use Virtualbox which allows you to run both your base
distro and then as many of the others as your hardware can handle as VMs?

This is how I go about investigating a new distro before committing to it.

[-- Attachment #2: Type: text/html, Size: 704 bytes --]

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-23, Mark Knecht <markknecht@gmail.com> wrote:
> On Fri, Feb 23, 2024 at 11:59 AM Grant Edwards <grant.b.edwards@gmail.com>
> wrote:
>>
>> The simple solution is to give up on multi-booting a dozen different
>> distros on a single disk and buy a pocketful of USB 3 thumb drives.
>>
>
> Given performance does drop a bit and there can be issues with allocating
> hardware, why not use Virtualbox which allows you to run both your base
> distro and then as many of the others as your hardware can handle as VMs?

The machine is used for testing PCI and PCI-express boards and their
drivers. I don't really trust PCI passthrough (especially when it
comes to interrupt handling) enought to do such testing in a
VM. IFAICT, it's very rare for customers to use VMs. If customers do
start to use VMs, then we'll have to test with VMs also.

--
Grant

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 1385 bytes --]

On Fri, Feb 23, 2024 at 12:52 PM Grant Edwards <grant.b.edwards@gmail.com>
wrote:
>
> On 2024-02-23, Mark Knecht <markknecht@gmail.com> wrote:
> > On Fri, Feb 23, 2024 at 11:59 AM Grant Edwards <
grant.b.edwards@gmail.com>
> > wrote:
> >>
> >> The simple solution is to give up on multi-booting a dozen different
> >> distros on a single disk and buy a pocketful of USB 3 thumb drives.
> >>
> >
> > Given performance does drop a bit and there can be issues with
allocating
> > hardware, why not use Virtualbox which allows you to run both your base
> > distro and then as many of the others as your hardware can handle as
VMs?
>
> The machine is used for testing PCI and PCI-express boards and their
> drivers. I don't really trust PCI passthrough (especially when it
> comes to interrupt handling) enought to do such testing in a
> VM. IFAICT, it's very rare for customers to use VMs. If customers do
> start to use VMs, then we'll have to test with VMs also.
>
In that specific use case I completely agree.

The only other idea I had was to install <new distro> to a different
disk and then use something like Clonezilla to move it to the partition
you want it in on your system.

While I suspect you were being sarcastic I do not think any solution
that involves a 'pocketful of USB 3 thumb drives' will be satisfying.

Wishing you luck,
Mark

[-- Attachment #2: Type: text/html, Size: 1834 bytes --]

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-23, Mark Knecht <markknecht@gmail.com> wrote:

> The only other idea I had was to install <new distro> to a different
> disk and then use something like Clonezilla to move it to the partition
> you want it in on your system.
>
> While I suspect you were being sarcastic I do not think any solution
> that involves a 'pocketful of USB 3 thumb drives' will be satisfying.

Actually (assuming the thumb drives are relaible) it probably would
work fine. That's more-or-less the typical solution that my colleagues
use -- though it's a drawer full of hard drives in caddies instead of
USB thumb drives. Back in the day when we supported a couple versions
of SCO, various BSDs, Novell Netware and Solaris, it required a fait
bit of drawer space.

For now, I guess I'll stick with the scheme I'm using but switch from
DOS disklabel and gap to GPT disklabel and BIOS boot partition. It
seems ugly, but it's managable with the help of a few shell scripts
that are stored in the parition that belongs to the master copy of
grub.

Re: [gentoo-user] How to set up drive with many Linux distros?

[gentoo-user]

Since this is a fairly custom task, I would approach it with a custom
solution.

- GPT 
- systemd-boot
- One /boot partition
- One BTRFS-on-LUKS partition (formatted using the distro with the
oldest kernel)
- {@root,@home,@var,@srv,@opt}-{distro1,distro2,distro3} subvolumes
- Potentially {@distro1,distro2,distro3}-{downloads,documents,pictures}
subvolumes, if there's a usecase for that
- Bootstrap all the distros manually. Arch and Gentoo do that by
default, debian land has debootstrap, not too familiar with rhel
universe, but at the end of the day everything's a file :)

You could also write a script that keeps all the distros up to date
from within whichever one you're currently booted by mounting
subvolumes to /mnt or wherever, chrooting in and running the update.

You could also replace systemd-boot with efibootmgr to use UEFI boot
directly, but I would advise doing that after the everything is working
correctly as it's much easier to experiment using a good old bootloader
edit function.

BTRFS requires _some_ maintenance, but imho it's reasonable - run a
deduplication and defrag job on a scheduling tool of your choice and
you're good to go!

$0.02

Re: [gentoo-user] How to set up drive with many Linux distros?

[eric]

On 2/26/24 04:57, gentoo-user@krasauskas.dev wrote:
> You could also write a script that keeps all the distros up to date
> from within whichever one you're currently booted by mounting
> subvolumes to /mnt or wherever, chrooting in and running the update.

To avoid grub not being able to point to a newly updated kernel on one 
of the OS's installed, I use a "custom.cfg" file in all my /boot/grub/ 
directories for each OS where the "linix" and "initrd" point to the 
symbolic links of the kernel and init files which point to the newly 
updated files on most major distributions like ubuntu, arch, suse, and 
debian. The name of the symbolic links stay the same over upgrades. It 
works great when using UUID to identify the partition that has root and 
I can always boot into any of the OS's installed no matter which one 
hijacked the MBR.


https://forums.linuxmint.com/viewtopic.php?t=315584

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-26, eric <eriic1@cox.net> wrote:
> On 2/26/24 04:57, gentoo-user@krasauskas.dev wrote:
>> You could also write a script that keeps all the distros up to date
>> from within whichever one you're currently booted by mounting
>> subvolumes to /mnt or wherever, chrooting in and running the update.
>
> To avoid grub not being able to point to a newly updated kernel on one 
> of the OS's installed, I use a "custom.cfg" file in all my /boot/grub/ 
> directories for each OS where the "linix" and "initrd" point to the 
> symbolic links of the kernel and init files which point to the newly 
> updated files on most major distributions like ubuntu, arch, suse, and 
> debian. The name of the symbolic links stay the same over upgrades. It 
> works great when using UUID to identify the partition that has root and 
> I can always boot into any of the OS's installed no matter which one 
> hijacked the MBR.

Except I generally have multiple kernels installed for each of the
distros, and need to be able to choose which kernel to boot. There are
also various other boot options (e.g. "safe mode") offered by some
distros that I occasionally need to use.

> https://forums.linuxmint.com/viewtopic.php?t=315584

Interesting article, thanks.

After reading up more on UEFI, it looks like that would be even more
work and more mess. So, there seem to be two options:

 1) Stick to the dual-stage chainloading scheme I'm using now (though
    I'll probably switch from DOS to GTP disklabel). That way after
    selecting which parition (distro) to boot, I get all the boot
    options normally offered by that distro's install. Installing a
    distro involves letting it install to MBR and BIOS-boot,
    installing grub manually to the root partition, then restoring MBR
    and BIOS-boot.

 2) Use a single master grub to boot any distro.
 
    I think I'd need to write my own OS-prober. All of the distros I
    care about seem to now be using grub2 now. Instead of looking for
    kernels and initrd images and adding them to the master grub.cfg,
    I would probe for grub.cfg files, and for each one found
    incorporate the entire set of choices in that .cfg file as a
    submenu in the main grub.cfg menu.

    I think that in order to generate the distro's grub.cfg files, I
    still have to allow the distros to install grub to the MBR and
    BIOS-boot (or to a second disk that I don't care about), then
    restoring MRB/BIOS-boot.
    

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[eric]

On 2/26/24 11:01, Grant Edwards wrote:
> On 2024-02-26, eric <eriic1@cox.net> wrote:
>> On 2/26/24 04:57, gentoo-user@krasauskas.dev wrote:
>>> You could also write a script that keeps all the distros up to date
>>> from within whichever one you're currently booted by mounting
>>> subvolumes to /mnt or wherever, chrooting in and running the update.
>>
>> To avoid grub not being able to point to a newly updated kernel on one
>> of the OS's installed, I use a "custom.cfg" file in all my /boot/grub/
>> directories for each OS where the "linix" and "initrd" point to the
>> symbolic links of the kernel and init files which point to the newly
>> updated files on most major distributions like ubuntu, arch, suse, and
>> debian. The name of the symbolic links stay the same over upgrades. It
>> works great when using UUID to identify the partition that has root and
>> I can always boot into any of the OS's installed no matter which one
>> hijacked the MBR.
> 
> Except I generally have multiple kernels installed for each of the
> distros, and need to be able to choose which kernel to boot. There are
> also various other boot options (e.g. "safe mode") offered by some
> distros that I occasionally need to use.
> 

I agree, using the custom.cfg file would not work if needing to boot 
different kernels of the same OS and those kernels were being updated.

Regards,
Eric

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-26, eric <eriic1@cox.net> wrote:

> I agree, using the custom.cfg file would not work if needing to boot 
> different kernels of the same OS and those kernels were being updated.

The simple answer is to quit wasting time trying to multi-boot like
that and just buy a dozen USB flash drives.

--
Grant

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Wol]

On 26/02/2024 20:51, Grant Edwards wrote:
> On 2024-02-26, eric <eriic1@cox.net> wrote:
> 
>> I agree, using the custom.cfg file would not work if needing to boot
>> different kernels of the same OS and those kernels were being updated.
> 
> The simple answer is to quit wasting time trying to multi-boot like
> that and just buy a dozen USB flash drives.
> 
And then, if USB isn't the default boot media, he might as well sort out 
UEFI boot, and multi-boot that way.

Cheers,
Wol

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-26, Wol <antlists@youngman.org.uk> wrote:
> On 26/02/2024 20:51, Grant Edwards wrote:
>
>> The simple answer is to quit wasting time trying to multi-boot like
>> that and just buy a dozen USB flash drives.
>> 
> And then, if USB isn't the default boot media, he might as well sort out 
> UEFI boot, and multi-boot that way.

Except that every time I've found a write-up about multibooting a lot
of Linux distros with UEFI, it turns out that it doesn't actually work
very well and is more work to maintain than what I'm doing now.

--
Grant

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 1550 bytes --]

On Mon, Feb 26, 2024 at 4:45 PM Grant Edwards <grant.b.edwards@gmail.com>
wrote:
>
> On 2024-02-26, Wol <antlists@youngman.org.uk> wrote:
> > On 26/02/2024 20:51, Grant Edwards wrote:
> >
> >> The simple answer is to quit wasting time trying to multi-boot like
> >> that and just buy a dozen USB flash drives.
> >>
> > And then, if USB isn't the default boot media, he might as well sort out
> > UEFI boot, and multi-boot that way.
>
> Except that every time I've found a write-up about multibooting a lot
> of Linux distros with UEFI, it turns out that it doesn't actually work
> very well and is more work to maintain than what I'm doing now.
>
> --
> Grant

I have no experience beyond three operating systems on a single machine
but if you grabbed just 2 or 3 USB flash drives then I would think you
could test it pretty easily. I believe the UEFI boot procedures are
storing a unique ID for the disk or the partition you are requesting. If you
have a unique ID that's different for each flash drive it would (hopefully)
find the one you're looking for which should be relatively simple.

I would suggest you use the boot ordering feature and make the
system hard drive last in the list. If no USB devices are plugged in
it would default to your system drive. If a flash drive is plugged in
it should find its ID and boot that first.

I do not know if, for instance, you had 20 different drives listed in
your BIOS whether it would be a lot slower to boot but you could
test that yourself.

Good luck,
Mark

[-- Attachment #2: Type: text/html, Size: 2008 bytes --]

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Peter Humphrey]

On Tuesday, 27 February 2024 00:12:07 GMT Mark Knecht wrote:

> I have no experience beyond three operating systems on a single machine
> but if you grabbed just 2 or 3 USB flash drives then I would think you
> could test it pretty easily. I believe the UEFI boot procedures are
> storing a unique ID for the disk or the partition you are requesting. If you
> have a unique ID that's different for each flash drive it would (hopefully)
> find the one you're looking for which should be relatively simple.
> 
> I would suggest you use the boot ordering feature and make the
> system hard drive last in the list. If no USB devices are plugged in
> it would default to your system drive. If a flash drive is plugged in
> it should find its ID and boot that first.
> 
> I do not know if, for instance, you had 20 different drives listed in
> your BIOS whether it would be a lot slower to boot but you could
> test that yourself.

My experience is that the BIOS will discard any boot devices that are not 
present at the time of booting. I don't know whether that's typical or just 
what I've found.

-- 
Regards,
Peter.

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-02-22, Grant Edwards <grant.b.edwards@gmail.com> wrote:

> For many years, I've used a hard drive on which I have 8-10 Linux
> distros installed -- each in a separate (single) partition.
>
> [...]
>
> Is there an easier way to do this?

After some additional studying of UEFI and boot managers like rEFInd,
I decided that my current approach was still the easiest method. I did
switch from DOS to GPT disklabel (I bricked my old drive tring to
update the firmware, so I had to start over anyway).

In case anybody is interested in the gory details, I documented my
scheme and the helper shell scripts at

  https://github.com/GrantEdwards/Linux-Multiboot

--
Grant

Re: [gentoo-user] Re: How to set up drive with many Linux distros?

[Michael]

[-- Attachment #1: Type: text/plain, Size: 2247 bytes --]

On Friday, 8 March 2024 23:24:02 GMT Grant Edwards wrote:
> On 2024-02-22, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> > For many years, I've used a hard drive on which I have 8-10 Linux
> > distros installed -- each in a separate (single) partition.
> > 
> > [...]
> > 
> > Is there an easier way to do this?
> 
> After some additional studying of UEFI and boot managers like rEFInd,
> I decided that my current approach was still the easiest method. I did
> switch from DOS to GPT disklabel (I bricked my old drive tring to
> update the firmware, so I had to start over anyway).
> 
> In case anybody is interested in the gory details, I documented my
> scheme and the helper shell scripts at
> 
>   https://github.com/GrantEdwards/Linux-Multiboot


Thank you Grant for taking time to share your clear and well structured write 
up, what with helpful scripts and all!  Its easy to follow and should help 
others, hopefully before they discover belatedly many distro installers can 
mess up a multiboot scheme, if they don't step in to keep things in check.

Perhaps I'm picking up on semantics, but shouldn't this sentence:

"... The gap between the DOS disklabel and the first partition"

read:

"The gap between the MBR and the first partition"?

I'm saying this because the MBR in sector 0 contains the bootstrap code (446 
bytes), the partition table (a.k.a. DOS disklabel 4x16 bytes) and the boot 
sector signature (2 bytes).  On MBR partitioned disks the core.img is stored 
after the MBR, in sector 1 onward.

Your next paragraph pointed out something which I hadn't considered at any 
length.  Namely, the installation of GRUB's boot.img in a MBR or VBR also 
hardcodes in a block list format the location of the first sector where the 
core.img is stored and more importantly, the physical position of this sector 
can be altered both by COW fs (and by the wear levelling firmware of flash 
storage devices).

I had assumed both the COW fs and/or the flash controller will in both cases 
translate any physical data position to the logical layer and presented this 
to inquiring software.  Have you actually tried using btrfs as a distro's root 
fs to see if the VBR installed GRUB boot.img will ever lose access to the 
core.img?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: How to set up drive with many Linux distros?

[Grant Edwards]

On 2024-03-10, Michael <confabulate@kintzios.com> wrote:

> Perhaps I'm picking up on semantics, but shouldn't this sentence:
>
> "... The gap between the DOS disklabel and the first partition"
>
> read:
>
> "The gap between the MBR and the first partition"?

Yes, thanks -- MBR is more accurate, I've changed that sentence.

> Your next paragraph pointed out something which I hadn't considered at any 
> length.  Namely, the installation of GRUB's boot.img in a MBR or VBR also 
> hardcodes in a block list format the location of the first sector where the 
> core.img is stored and more importantly, the physical position of this sector 
> can be altered both by COW fs (and by the wear levelling firmware of flash 
> storage devices).
>
> I had assumed both the COW fs and/or the flash controller will in
> both cases translate any physical data position to the logical layer
> and presented this to inquiring software.  Have you actually tried
> using btrfs as a distro's root fs to see if the VBR installed GRUB
> boot.img will ever lose access to the core.img?

No, I haven't. I agree that the flash controller can't change the
logical address of a filesystem data block without the knowledge of the
filesystem, so I don't think controller layer wear-leveling would be
a problem. But, the filesystem layer is allowed to move data blocks
around, so flash-aware filesystems that attempt to do wear-leveling
or defragmentation could move data blocks.

Some of the descriptions I've read of "fancier" filesystem internals
have also implied implied that does happen under certain conditions,
but I may have misunderstood or the descriptions may have been wrong.

My use of these multi-boot installs have no need for anything beyond
exnN, so I've never tried using block lists with anything other than
extN filesystems. Since I am confident extN filesystems won't cause
problems, I've always stuck with that.

--
Grant