[gentoo-user] switch from gnome/systemd to xfce/openrc borked my system

[gentoo-user] switch from gnome/systemd to xfce/openrc borked my system

[Raffaele Belardi]

[-- Attachment #1: Type: text/plain, Size: 1968 bytes --]

Yesterday I tried to switch my ~amd64 box from Gnome/systemd to Xfce/openrc. I followed 
the wiki [1], [2] to install Xfce from a Gnome terminal:

- switch profile from 17.1/desktop/gnome/systemd to 17.1/desktop
- emerge xfce4-meta and some xfce4 applications/panels/extras
- unmerged systemd and emerged OpenRC
- emerge -uDvN world to account for the different profile flags

During the emerge I had to hard reset the system [3] which obviously did not boot so I 
found a PCLinuxOS live cd from 2014 and managed to chroot into the partially updated 
system. I resumed the emerge successfully, unmerged gnome and dependencies (this almost 
took more than building Xfce...), rebuilt the kernel with the init system set to OpenRC 
(make && make install), update the grub menu to no longer pass to the the kernel the 
systemd init.

Now both emerge -uDvN world and emerge --depclean show a clean system. But the system 
won't boot...

I attach a picture of the last kernel messages. I am not familiar with init systems but it 
seems to me that init has started but then hangs at some point. From the chroot I 
re-checked the /etc/runlevels contents, all links seem fine except a couple of stale ones 
which I removed (mpd, syklogd and few others from pre-Gnome days). /etc/inititab looks 
fine also.

One thing I notice from the boot log is that the root FS requires recovery. My live CDs 
did not let me because they are too old  so I'll try to find a more up to date live CD.

Other ideas? Have I missed some obvious step in the process?

thanks,

raffaele


[1] https://wiki.gentoo.org/wiki/Xfce
[2] https://wiki.gentoo.org/wiki/Xfce/Guide
[3] While emerge was ongoing I left the system alone for dinner time, when I got back gdm 
had kicked in and did not let me log back in, neither through the GUI nor in a terminal 
with CTRL-ALT-Fx. This is probably due to me having unmerged GDM before starting the 
emerge to fix some dependency, thus a vivid example of PEBCAK

[-- Attachment #2: IMG_20190818_220947.jpg --]
[-- Type: image/jpeg, Size: 179383 bytes --]

[gentoo-user] Re: switch from gnome/systemd to xfce/openrc borked my system

[Raffaele Belardi]

Raffaele Belardi wrote:
> 
> One thing I notice from the boot log is that the root FS requires recovery. My live CDs 
> did not let me because they are too old  so I'll try to find a more up to date live CD.
> 

Looking better at the kernel log, for sda1 (the root partition) it says: "Recovery 
complete" so I don't think a new live CD will help. I'm really out of ideas.

raffaele

Re: [gentoo-user] Re: switch from gnome/systemd to xfce/openrc borked my system

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1753 bytes --]

On Monday, 19 August 2019 07:41:20 BST Raffaele Belardi wrote:
> Raffaele Belardi wrote:
> > One thing I notice from the boot log is that the root FS requires
> > recovery. My live CDs did not let me because they are too old  so I'll
> > try to find a more up to date live CD.
> Looking better at the kernel log, for sda1 (the root partition) it says:
> "Recovery complete" so I don't think a new live CD will help. I'm really
> out of ideas.
> 
> raffaele

You have 3 drives attached while you're trying to boot.  The kernel seems to 
come to a stop after /dev/sdc.  It may need some driver for this device/fs.  
I'd start by unplugging any drives which do not contain the system you're 
trying to boot, then go through a step by step process of installing/setting 
up openrc, DM and boot loader.

The DM is not necessary to boot your system, but while you chrooted into it 
you might as well install and set up sddm as a DM - there are others but be 
careful they do not try to bring in 2/3 of Gnome and its dependencies too.

Re-install GRUB or whichever boot manager you use and make sure it points to 
the correct kernel.  If you're on an UEFI system and you boot directly using 
the kernel EFI stub, re-run efibootmgr to specify the kernel UEFI will boot 
with, but first run fsck.vfat on the EFI partition just in case this fs was 
messed up too.

Make sure you are using a kernel set up for openrc.

In /etc/rc.conf set up a log file and temporarily enable logging.  If any 
openrc scripts fail and can't boot, you will able to look at the logs when you 
chroot back into it - using less/cat/plain text editor.  ;-)

I hope the above should allow you to boot, or at least arrive at some 
meaningful failure message to resolve.
-- 
Regards,

Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] switch from gnome/systemd to xfce/openrc borked my system

[David Haller]

Hello,

On Mon, 19 Aug 2019, Raffaele Belardi wrote:
[..]
>During the emerge I had to hard reset the system [3] which obviously did not
>boot so I found a PCLinuxOS live cd from 2014 and managed to chroot into the
>partially updated system. I resumed the emerge successfully, unmerged gnome
>and dependencies (this almost took more than building Xfce...), rebuilt the
>kernel with the init system set to OpenRC (make && make install),

Did you also install the modules? (make modules_install)
Did you update the initrd?

Why not use 'genkernel'? The default-config should work in your case,
but you might look at the config anyway (/etc/genkernel.conf), e.g. at
the BOOTLOADER variable. Then 'cd' to your kernel-sources-dir and run
'genkernel --kerneldir=. all'.

HTH,
-dnh

-- 
You know, I found myself right where I left me...

Re: [gentoo-user] Re: switch from gnome/systemd to xfce/openrc borked my system

[Raffaele Belardi]

Mick wrote:
> On Monday, 19 August 2019 07:41:20 BST Raffaele Belardi wrote:
> 
> You have 3 drives attached while you're trying to boot.  The kernel seems to
> come to a stop after /dev/sdc.  It may need some driver for this device/fs.
> I'd start by unplugging any drives which do not contain the system you're
> trying to boot, then go through a step by step process of installing/setting
> up openrc, DM and boot loader.

sdc is an external USB drive, I'll try to unplug that.

> The DM is not necessary to boot your system, but while you chrooted into it
> you might as well install and set up sddm as a DM - there are others but be
> careful they do not try to bring in 2/3 of Gnome and its dependencies too.

I'll do but first I want to see a working terminal, too much stuff to debug otherwise.

> Re-install GRUB or whichever boot manager you use and make sure it points to
> the correct kernel.  If you're on an UEFI system and you boot directly using
> the kernel EFI stub, re-run efibootmgr to specify the kernel UEFI will boot
> with, but first run fsck.vfat on the EFI partition just in case this fs was
> messed up too.

It's grub2, non-UEFI. I don't normally reinstall it when I update the kernel, I only run 
grub-mkconfig. I did the same this time.

> Make sure you are using a kernel set up for openrc.

Good catch, although I'm not sure where to find that info in the available kernel log. 
I'll look better, I need to stop it from scrolling.

> In /etc/rc.conf set up a log file and temporarily enable logging.  If any
> openrc scripts fail and can't boot, you will able to look at the logs when you
> chroot back into it - using less/cat/plain text editor.  ;-)

Good idea.

> I hope the above should allow you to boot, or at least arrive at some
> meaningful failure message to resolve.
> 

One of the last things printed in the kernel log is "random: crng init done". The random 
service is part (possibly the last service) of the boot runlevel which is entered after 
the sysinit runlevel. So apparently a lot of openrc stuff has already started 
successfully. Instead, nothing from the default runlevel is output. I'll re-check those 
services.

raffaele

Re: [gentoo-user] switch from gnome/systemd to xfce/openrc borked my system

[Raffaele Belardi]

David Haller wrote:
> Hello,
> 
> On Mon, 19 Aug 2019, Raffaele Belardi wrote:
> [..]
>> During the emerge I had to hard reset the system [3] which obviously did not
>> boot so I found a PCLinuxOS live cd from 2014 and managed to chroot into the
>> partially updated system. I resumed the emerge successfully, unmerged gnome
>> and dependencies (this almost took more than building Xfce...), rebuilt the
>> kernel with the init system set to OpenRC (make && make install),
> 
> Did you also install the modules? (make modules_install)
> Did you update the initrd?

No modules here, everything built in. Also no initrd.

> Why not use 'genkernel'? The default-config should work in your case,
> but you might look at the config anyway (/etc/genkernel.conf), e.g. at
> the BOOTLOADER variable. Then 'cd' to your kernel-sources-dir and run
> 'genkernel --kerneldir=. all'.

I've always compiled the kernel directly so I am not familiar with genkernel. But I'll 
look again into my kernel's config and make sure it is the right kernel that's being loaded.

raffaele

Re: [gentoo-user] switch from gnome/systemd to xfce/openrc borked my system

[Rich Freeman]

On Mon, Aug 19, 2019 at 2:29 AM Raffaele Belardi
<raffaele.belardi@st.com> wrote:
>
> Yesterday I tried to switch my ~amd64 box from Gnome/systemd to Xfce/openrc. I followed
> the wiki [1], [2] to install Xfce from a Gnome terminal:
>
> - switch profile from 17.1/desktop/gnome/systemd to 17.1/desktop
> - emerge xfce4-meta and some xfce4 applications/panels/extras
> - unmerged systemd and emerged OpenRC
> - emerge -uDvN world to account for the different profile flags

Next time you do something like this, keep in mind that Gnome and xfce
can co-exist on the same system, and so can openrc and systemd.

At this point you're probably just going to want to troubleshoot what
you are left with, though you could consider reverting back to your
old config and starting over if you have backups/etc.

I imagine that not many people move from systemd to openrc, since the
latter is basically the default on Gentoo already.  If I were going to
migrate a working system between the two I would probably do it
stepwise:

1.  Rebuild the kernel with support for both systemd and openrc.  Boot
that (under systemd) and confirm it is working.
2.  Install xfce and get that working fine (under systemd).  That
really has no tie-in to the service manager but if you have this
working it is one less thing to mess with and it simplifies your
system.
3.  Install openrc and reboot under systemd just to make sure
everything is still working fine.  I forget what the defaults are but
you might need to tweak your systemd USE flags so that it uses the
sysvinit versions of halt/reboot/poweroff/telinit and so on.  It works
just fine with its own version of these tools or with sysvinit.
4.  Make sure you have your openrc configured the way you want it (I
don't think it has any issues with using rc-update and so on while
systemd is running, but I haven't tried that).
5.  Switch your kernel command line to boot with openrc, and make a
note of what it said before.  If it boots fine you're now running
openrc and just have to clean up the stuff you don't want.  If it
doesn't boot you just have to edit your command line and you're back
up and running with systemd until you sort it out.
6.  Switch your profile, do the -uDvN to rebuild anything impacted,
and depclean the stuff you don't need.  Reboot to test.

By doing it this way you will be just making one change at a time with
a reboot in-between so that you know what broke if something breaks.
The way you did it is potentially more time-efficient, but if
something breaks you are going to be hunting to figure out what it
was.  Since all the packages you're changing are capable of
co-existing there is no reason to switch cold turkey.

Now, on a new install or a host I didn't care so much about uptime for
I'd probably do it your way, and just revert to a backup.  In a
production environment where reboots are a concern I'd be working out
the procedure on a test host.

Oh, yeah, and step 0 is to make a backup...  :)

-- 
Rich

Re: [gentoo-user] Re: switch from gnome/systemd to xfce/openrc borked my system

[Daniel Frey]

On 8/19/19 5:24 AM, Raffaele Belardi wrote:
> 
>> Make sure you are using a kernel set up for openrc.
> 
> Good catch, although I'm not sure where to find that info in the 
> available kernel log. I'll look better, I need to stop it from scrolling.
> 

Did you update grub and remove the init= line that starts systemd?

>> In /etc/rc.conf set up a log file and temporarily enable logging.  If any
>> openrc scripts fail and can't boot, you will able to look at the logs 
>> when you
>> chroot back into it - using less/cat/plain text editor.  ;-)
> 
> Good idea.
> 
>> I hope the above should allow you to boot, or at least arrive at some
>> meaningful failure message to resolve.
>>
> 
> One of the last things printed in the kernel log is "random: crng init 
> done". The random service is part (possibly the last service) of the 
> boot runlevel which is entered after the sysinit runlevel. So apparently 
> a lot of openrc stuff has already started successfully. Instead, nothing 
> from the default runlevel is output. I'll re-check those services.
> 

Hmm, is it possible that it's waiting for entropy? Try moving the mouse 
like a madman for 20 seconds or so.

Dan

Re: [gentoo-user] Re: switch from gnome/systemd to xfce/openrc borked my system

[Peter Humphrey]

On Monday, 19 August 2019 13:24:05 BST Raffaele Belardi wrote:
> Mick wrote:
> > On Monday, 19 August 2019 07:41:20 BST Raffaele Belardi wrote:
> > 
> > You have 3 drives attached while you're trying to boot.  The kernel seems
> > to come to a stop after /dev/sdc.  It may need some driver for this
> > device/fs. I'd start by unplugging any drives which do not contain the
> > system you're trying to boot, then go through a step by step process of
> > installing/setting up openrc, DM and boot loader.
> 
> sdc is an external USB drive, I'll try to unplug that.
> 
> > The DM is not necessary to boot your system, but while you chrooted into
> > it
> > you might as well install and set up sddm as a DM - there are others but
> > be
> > careful they do not try to bring in 2/3 of Gnome and its dependencies too.
> 
> I'll do but first I want to see a working terminal, too much stuff to debug
> otherwise.
> > Re-install GRUB or whichever boot manager you use and make sure it points
> > to the correct kernel.  If you're on an UEFI system and you boot directly
> > using the kernel EFI stub, re-run efibootmgr to specify the kernel UEFI
> > will boot with, but first run fsck.vfat on the EFI partition just in case
> > this fs was messed up too.
> 
> It's grub2, non-UEFI. I don't normally reinstall it when I update the
> kernel, I only run grub-mkconfig. I did the same this time.
> 
> > Make sure you are using a kernel set up for openrc.
> 
> Good catch, although I'm not sure where to find that info in the available
> kernel log. I'll look better, I need to stop it from scrolling.

It may be possible to hit CTRL-s to pause the scrolling, then CTRL-q to resume 
it.

> > In /etc/rc.conf set up a log file and temporarily enable logging.  If any
> > openrc scripts fail and can't boot, you will able to look at the logs when
> > you chroot back into it - using less/cat/plain text editor.  ;-)
> 
> Good idea.
> 
> > I hope the above should allow you to boot, or at least arrive at some
> > meaningful failure message to resolve.
> 
> One of the last things printed in the kernel log is "random: crng init
> done". The random service is part (possibly the last service) of the boot
> runlevel which is entered after the sysinit runlevel. So apparently a lot
> of openrc stuff has already started successfully. Instead, nothing from the
> default runlevel is output. I'll re-check those services.
> 
> raffaele


-- 
Regards,
Peter.

Re: [gentoo-user] Re: switch from gnome/systemd to xfce/openrc borked my system

[Raffaele Belardi]

Daniel Frey wrote:
> On 8/19/19 5:24 AM, Raffaele Belardi wrote:
>>
>>> Make sure you are using a kernel set up for openrc.
>>
>> Good catch, although I'm not sure where to find that info in the available kernel log. 
>> I'll look better, I need to stop it from scrolling.
>>
> 
> Did you update grub and remove the init= line that starts systemd?

Yes.

>> One of the last things printed in the kernel log is "random: crng init done". The random 
>> service is part (possibly the last service) of the boot runlevel which is entered after 
>> the sysinit runlevel. So apparently a lot of openrc stuff has already started 
>> successfully. Instead, nothing from the default runlevel is output. I'll re-check those 
>> services.
>>
> 
> Hmm, is it possible that it's waiting for entropy? Try moving the mouse like a madman for 
> 20 seconds or so.

I read on the internet about the crng delay issue. I don't think this is the case because 
it hangs after it has finished the random rc script, but I'll give it a try.

raffaele

Re: [gentoo-user] switch from gnome/systemd to xfce/openrc borked my system

[Raffaele Belardi]

Rich Freeman wrote:
> Next time you do something like this, keep in mind that Gnome and xfce
> can co-exist on the same system, and so can openrc and systemd.

Good point, I did not know, in particular for the init systems I thought it was exactly 
the opposite.

> At this point you're probably just going to want to troubleshoot what
> you are left with, though you could consider reverting back to your
> old config and starting over if you have backups/etc.

Backups? I'm a software developer, I can't afford spending time making backups (just kidding).
At this point of the boot process the system did very few things, the problem should be 
relatively easy to trace so I refuse to give up. Long nights ahead.

> I imagine that not many people move from systemd to openrc, since the
> latter is basically the default on Gentoo already.  If I were going to
> migrate a working system between the two I would probably do it
> stepwise:

Ha, this is the HOWTO I was looking for yesterday! Oh well, it'll be for next time.

> Now, on a new install or a host I didn't care so much about uptime for
> I'd probably do it your way, and just revert to a backup.  In a

Not really production environment but I have to fix it before the other users (the kids) 
come back from vacation or my reputation will quickly sink!

raffaele

Re: [gentoo-user] switch from gnome/systemd to xfce/openrc borked my system

[Rich Freeman]

On Mon, Aug 19, 2019 at 10:42 AM Raffaele Belardi
<raffaele.belardi@st.com> wrote:
>
> Rich Freeman wrote:
> > Next time you do something like this, keep in mind that Gnome and xfce
> > can co-exist on the same system, and so can openrc and systemd.
>
> Good point, I did not know, in particular for the init systems I thought it was exactly
> the opposite.
>

The only area of incompatibility I'm aware of are the
sysvinit-compatibility links.  Both sysvinit and systemd provide
implementations of common utilities like poweroff, halt, reboot,
telinit, and so on.  There is also init itself.

The versions that come with sysvinit are compatible with both sysvinit
and systemd.  If you don't have sysvinit then systemd can supply
these.  Systemd itself doesn't require these utilities but they are
useful both for compatibility and convenience.  (ie "systemctl
poweroff" works fine, as does sending the command via dbus, but
scripts or sysadmins might prefer to be able to just run "poweroff").
The versions of these supplied by systemd are not compatible with
sysvinit.

A USE flag toggles whether systemd installs these utilities.  If it
does then it blocks sysvinit.  So, you just have to switch that USE
flag to install the two in parallel.  If you don't have systemd
install "init" then you do need to have a kernel command line to
launch systemd directly as init.

Offhand I think that is really the only conflict between the two.
Systemd doesn't use anything but those compatibility utils from
sysvinit but it doesn't mind them being around, and nothing in
sysvinit/openrc should even notice that systemd is installed.

As long as you set the USE flag appropriately you can dual-boot
between the two very easily.  The only gotcha is keeping all your
configs up-to-date as openrc and systemd store things in different
places.  When you install systemd it takes a snapshot of many of your
openrc settings but that is a one-time operation.  Some of those
settings are hard to change if systemd isn't running as PID 1 - I
think the wiki has instructions for how to do this.

--
Rich

Re: [gentoo-user] switch from gnome/systemd to xfce/openrc borked my system - I GIVE UP

[Raffaele Belardi]

Raffaele Belardi wrote:
> At this point of the boot process the system did very few things, the problem should be 
> relatively easy to trace so I refuse to give up. Long nights ahead.

And at this point of the day I give up.

My findings:
- the 'random: crng init done' printed during the boot is the kernel, not OpenRC. The 
OpenRC script just manages the seed and does not print that string. So, most probably 
OpenRC is never started.
- from the chroot I invoked /sbin/openrc <runlevel>. It spits out errors due to the chroot 
environment but other than that seems to work fine for all runlevels. So the init scripts 
and /sbin/openrc are fine.
- I also issued startxfce4 from the chroot and got a 'working' GUI. Not very useful but I 
was curious.
- one of the init scripts complained that it was unable to open /run and actually the 
directory was not present. Possibly it was deleted as a side effect of the system crash? 
Anyway, I restored it but no luck.
- I grub-loaded a different kernel, one built for systemd. It stops in the exact same 
place as the openrc-built one.
- I recompiled the kernel with openrc and systemd options and made sure it was loaded 
(renamed the image file, got grub error, renamed it back, grub happy). I increased the 
kernel (I think console message) log level but nothing useful was output.
- /sbin/init is the next possible failure point but without logs there's not much to 
debug. I re-emerged sysvinit and openrc but no go.

Ok, I've had enough. Thanks to all who contributed, this issue will remain a mystery.

raffaele

[gentoo-user] Re: switch from gnome/systemd to xfce/openrc borked my system - I GIVE UP

[Nuno Silva]

On 2019-08-20, Raffaele Belardi wrote:

> Raffaele Belardi wrote:
[...]
> - I grub-loaded a different kernel, one built for systemd. It stops in
> the exact same place as the openrc-built one.

What are the kernel command lines for both kernels?

-- 
Nuno Silva

Re: [gentoo-user] Re: switch from gnome/systemd to xfce/openrc borked my system - I GIVE UP

[Raffaele Belardi]

Nuno Silva wrote:
> On 2019-08-20, Raffaele Belardi wrote:
> 
>> Raffaele Belardi wrote:
> [...]
>> - I grub-loaded a different kernel, one built for systemd. It stops in
>> the exact same place as the openrc-built one.
> 
> What are the kernel command lines for both kernels?
> 

I'm currently on a different system so, from the top of my head:

For openrc something like "root=/dev/sda1 ro iommu=soft"
For systemd "root=/dev/sda1 ro init=/path/to/systemd/init iommu=soft"

iommu is required to workaround an USB3 issue on my motherboard.

raffaele