[gentoo-user] Emerge -u -k package install order - broken system

[gentoo-user] Emerge -u -k package install order - broken system

[Alexander Puchmayr]

Hi there,

I just tried to upgrade a older installation via binary packages and this 
broke my system. After around 25 packages of almost 300 it stopped with error 
and failing packages.

$ emerge
Failed to validate a sane '/dev'.
bash process substitution doesn't work; this may be an indication of a broken 
'/dev/fd'.
$ ls -l /dev/fd/
insgesamt 0
lrwx------ 1 root root 64  6. Sep 14:18 0 -> /dev/pts/0
lrwx------ 1 root root 64  6. Sep 14:18 1 -> /dev/pts/0
lrwx------ 1 root root 64  6. Sep 14:18 2 -> /dev/pts/0
lr-x------ 1 root root 64  6. Sep 14:18 3 -> /proc/27261/fd

--> looks allright, but:

$ bash
bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /lib64/
libreadline.so.8)

--> system broken(!), cannot start any shell anymore, cannot install anything 
anymore and it's obvious that the system is bricked after reboot or even when 
the ssh session I'm logged in is closed.

It seems like as if sys-libs/readline-8.1_p1-r1-1:0/8::gentoo is installed 
*before* installing a suitable glibc, breaking any binary that has the useflag 
readline (including bash).

Two questions:
How do I get out of this mess? 
Why does portage not work in correct package order? Portage bug?

Re: [gentoo-user] Emerge -u -k package install order - broken system

[Jack]

On 2021.09.06 10:33, Alexander Puchmayr wrote:
> Hi there,
> 
> I just tried to upgrade a older installation via binary packages and  
> this
> broke my system. After around 25 packages of almost 300 it stopped  
> with error
> and failing packages.
> 
> $ emerge
> Failed to validate a sane '/dev'.
> bash process substitution doesn't work; this may be an indication of  
> a broken
> '/dev/fd'.
> $ ls -l /dev/fd/
> insgesamt 0
> lrwx------ 1 root root 64  6. Sep 14:18 0 -> /dev/pts/0
> lrwx------ 1 root root 64  6. Sep 14:18 1 -> /dev/pts/0
> lrwx------ 1 root root 64  6. Sep 14:18 2 -> /dev/pts/0
> lr-x------ 1 root root 64  6. Sep 14:18 3 -> /proc/27261/fd
> 
> --> looks allright, but:
> 
> $ bash
> bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by  
> /lib64/
> libreadline.so.8)
> 
> --> system broken(!), cannot start any shell anymore, cannot install  
> anything
> anymore and it's obvious that the system is bricked after reboot or  
> even when
> the ssh session I'm logged in is closed.
> 
> It seems like as if sys-libs/readline-8.1_p1-r1-1:0/8::gentoo is  
> installed
> *before* installing a suitable glibc, breaking any binary that has  
> the useflag
> readline (including bash).
> 
> Two questions:
> How do I get out of this mess?
> Why does portage not work in correct package order? Portage bug?
It might help if you stated which version of packages you currently  
have installed - specifically glibc.

As for recovery, you most likely need to boot to a live image (CD or  
USB) then chroot into the existing system.  I'm only guessing as to  
what is the minimal list of files you will need to replace, but I'd see  
if you can find or create a binary package of the latest glibc and then  
install or unpack that to your system.

As to whether this might be a bug in portage, I'd say you need to  
provide more details about exactly what you did.  What emerge line,  
what you mean by installation via binary packages, and where you got  
those binary packages.

If you do still have a running shell, does emerge still run at all?

Re: [gentoo-user] Emerge -u -k package install order - broken system

[Alexander Puchmayr]

Am Montag, 6. September 2021, 19:38:38 CEST schrieb Jack:
> On 2021.09.06 10:33, Alexander Puchmayr wrote:
> > Hi there,
> > 
> > I just tried to upgrade a older installation via binary packages and
> > this
> > broke my system. After around 25 packages of almost 300 it stopped
> > with error
> > and failing packages.
> > 
> > $ emerge
> > Failed to validate a sane '/dev'.
> > bash process substitution doesn't work; this may be an indication of
> > a broken
> > '/dev/fd'.
> > $ ls -l /dev/fd/
> > insgesamt 0
> > lrwx------ 1 root root 64  6. Sep 14:18 0 -> /dev/pts/0
> > lrwx------ 1 root root 64  6. Sep 14:18 1 -> /dev/pts/0
> > lrwx------ 1 root root 64  6. Sep 14:18 2 -> /dev/pts/0
> > lr-x------ 1 root root 64  6. Sep 14:18 3 -> /proc/27261/fd
> > 
> > --> looks allright, but:
> > 
> > $ bash
> > bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by
> > /lib64/
> > libreadline.so.8)
> > 
> > --> system broken(!), cannot start any shell anymore, cannot install
> > anything
> > anymore and it's obvious that the system is bricked after reboot or
> > even when
> > the ssh session I'm logged in is closed.
> > 
> > It seems like as if sys-libs/readline-8.1_p1-r1-1:0/8::gentoo is
> > installed
> > *before* installing a suitable glibc, breaking any binary that has
> > the useflag
> > readline (including bash).
> > 
> > Two questions:
> > How do I get out of this mess?
> > Why does portage not work in correct package order? Portage bug?
> 
> It might help if you stated which version of packages you currently
> have installed - specifically glibc.
> 
> As for recovery, you most likely need to boot to a live image (CD or
> USB) then chroot into the existing system.  I'm only guessing as to
> what is the minimal list of files you will need to replace, but I'd see
> if you can find or create a binary package of the latest glibc and then
> install or unpack that to your system.
> 
> As to whether this might be a bug in portage, I'd say you need to
> provide more details about exactly what you did.  What emerge line,
> what you mean by installation via binary packages, and where you got
> those binary packages.
> 
> If you do still have a running shell, does emerge still run at all?

What I did is:
* On my build host, I compiled the full system and creating packages
* On the client (i.e. The machine in question) I simply did a emerge -auvDNk 
world.
* Portage did not complain about any conflicts, the amount of packages is what 
I expected, so I started the upgrade.
* After about 25 packages installed successfully, emerge stopped with the 
errors explained before.

For me the problem is solved now as I could recover the system, but I still 
would like to know how to prevent this -- installing glibc prior world may be 
a good idea.

Thanks,
  Alex

Re: [gentoo-user] Emerge -u -k package install order - broken system [SOLVED]

[Alexander Puchmayr]

Am Montag, 6. September 2021, 16:33:32 CEST schrieb Alexander Puchmayr:
> Hi there,
> 
> I just tried to upgrade a older installation via binary packages and this
> broke my system. After around 25 packages of almost 300 it stopped with
> error and failing packages.
> 
> $ emerge
> Failed to validate a sane '/dev'.
> bash process substitution doesn't work; this may be an indication of a
> broken '/dev/fd'.
> $ ls -l /dev/fd/
> insgesamt 0
> lrwx------ 1 root root 64  6. Sep 14:18 0 -> /dev/pts/0
> lrwx------ 1 root root 64  6. Sep 14:18 1 -> /dev/pts/0
> lrwx------ 1 root root 64  6. Sep 14:18 2 -> /dev/pts/0
> lr-x------ 1 root root 64  6. Sep 14:18 3 -> /proc/27261/fd
> 
> --> looks allright, but:
> 
> $ bash
> bash: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by /lib64/
> libreadline.so.8)
> 
> --> system broken(!), cannot start any shell anymore, cannot install
> anything anymore and it's obvious that the system is bricked after reboot
> or even when the ssh session I'm logged in is closed.
> 
> It seems like as if sys-libs/readline-8.1_p1-r1-1:0/8::gentoo is installed
> *before* installing a suitable glibc, breaking any binary that has the
> useflag readline (including bash).
> 
> Two questions:
> How do I get out of this mess?
> Why does portage not work in correct package order? Portage bug?

The problem was caused by a newly compiled sys-libs/readline as binary 
package. The package itself was compiled on my buildhost by a emerge -auvDN 
world, and when readline was compiled, the latest glibc-2.33 was already 
installed on the buildhost, hence the readline packet had a (implicit) 
dependency to >=glibc-33. Portage does not seem to be aware of that (the glibc 
version is not a dependency of readline), so portage did not care and install 
it at some suitable position later. From the time on that readline was emerged 
from that binary package, all programs linked with it have now that implicit 
dependency of >=glibc-2.33, which is not yet installed (which was 2.28). Bang!

HOW TO FIX THIS BREAKAGE:

For the system recovery I was lucky to have a backup of the old packages, so I 
could extract the readline-tbz2-package file to the temporary directly and copy 
the so files to /usr/local/lib64 (I chose /usr/local/ to avoid overwriting the 
package files) 

Ls -l /usr/local/lib64/
insgesamt 368
lrwxrwxrwx 1 root root     15  6. Sep 14:53 libhistory.so -> libhistory.so.8
lrwxrwxrwx 1 root root     17  6. Sep 15:04 libhistory.so.8 -> libhistory.so.
8.0
-rwxr-xr-x 1 root root  47368  6. Sep 14:53 libhistory.so.8.0
lrwxrwxrwx 1 root root     16  6. Sep 14:53 libreadline.so -> libreadline.so.8
lrwxrwxrwx 1 root root     18  6. Sep 15:04 libreadline.so.8 -> 
libreadline.so.8.0
-rwxr-xr-x 1 root root 325464  6. Sep 14:53 libreadline.so.8.0

Then, I created a ld.so.conf config file to ensure that /usr/local/lib is at 
*first* place:

echo "/usr/local/lib64" >/etc/ld.so.conf.d/10-temporary-fix.conf 
ldconfig

Notes: 
* The config file *must* end with .conf
* Although /usr/local/lib64 is already in the ld.so.conf file, this step is 
necessary to get it at first place; otherwise it is after /lib64 and /usr/
lib64, and ld.so finds the non-working version first
* setting LD_LIBRARY_PATH does not work as ebuild does not forward the 
environment variables from the calling process.

Test it with ldd /usr/bin/bash -- no errors shall be reported, and the 
readline library shall show the /usr/local/lib64 path.

Now, I could emerge glibc-2.33 the usual way (emerge -avk1 glibc), and then 
afterwards remove the temporary ld.so.conf file

rm /etc/ld.so.conf.d/10-temporary-fix.conf 
ldconfig

Test again with  ldd /usr/bin/bash -- the new readline library from /usr/lib64 
and /lib64 shall now be used. 
Now, the libraries from /usr/local/lib can be removed safely.

System recovered!

Best regards,
 Alex