[gentoo-user] system stopped booting

[gentoo-user] system stopped booting

[Michael George]

I'm having a strange problem with a system that we're putting together
to be an LDAP-based PDC.

I had it all configured and it was working fine.  I had to put the
project aside for a couple weeks as I had other things to work on and
then I came back to it.  Since it had been a while, I updated the whole
system, updated the configs with etc-update, and I'm pretty sure I
rebooted to make sure all was well.

About a week later, I booted it and it wouldn't boot.  It got stuck at
"configuring system to use udev" and never gets okay.  I just sits there
for ever.

I booted a gentoo live cd as a rescue and found that I could chroot into
the broken system and work with it.  I tried to rebuild the whole thing,
but some emerges (one I remember is "udev") would stall.  They will
only get so far as the initial ">>> emerge" line and they will sit there
for ever.

The emerges that fail will always fail.  The emerges that succeed will
do so build after build.

I ran memtest on the RAM and it failed some tests so I thought I'd found
the problem.  A couple weeks later, I have new RAM for it, but it
behaves the same.

I am not sure where to go with this one.  I'm not sure why it won't
build or boot.  The system had been running just fine before the update
and I *think* it booted after it.  However, I realize I might have
dorked some config file in /etc, but I am not sure which it might be...

If anyone has any suggestions, I'd like to find a way out of this other
than reinstalling the system.  Such a thing might happen again in the
future to a non-development system and I'd like to know the way to get it
working again.

Thank you.

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Arturo 'Buanzo' Busleiman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael George wrote:
> If anyone has any suggestions, I'd like to find a way out of this other
> than reinstalling the system.  Such a thing might happen again in the
> future to a non-development system and I'd like to know the way to get it
> working again.

Any kernel messages or syslog entries during the chrooted work?

- --
Arturo "Buanzo" Busleiman - www.buanzo.com.ar
Consultor en Seguridad Informatica / Dominio Digital TV - Da FOSS man!
KTP Consultores - info AT ktpconsultores.com.ar

Romper un sistema de seguridad los acerca tanto a ser hackers como el
encender autos puenteando los convierte en ingenieros automotrices.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFDlKMfAlpOsGhXcE0RArDuAJ9/jYxi4X80VTV+mqz2y00F36J6VgCcDzY/
e6fbLkr8Q8gm1ytrnrX0epw=
=0cNx
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[b.n.]

Have you checked the mobo? Memtest will report errors if the mobo is bad.

m.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Michael George]

On Mon, Dec 05, 2005 at 09:51:27PM +0000, b.n. wrote:
> Have you checked the mobo? Memtest will report errors if the mobo is bad.

I have new RAM in it and it passed a full course of memtest86 with no problems
reported.

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Michael George]

On Mon, Dec 05, 2005 at 05:29:19PM -0300, Arturo 'Buanzo' Busleiman wrote:
> Michael George wrote:
> > If anyone has any suggestions, I'd like to find a way out of this other
> > than reinstalling the system.  Such a thing might happen again in the
> > future to a non-development system and I'd like to know the way to get it
> > working again.
> 
> Any kernel messages or syslog entries during the chrooted work?

Embarassingly, I didn't think to check that...  I am out with meetings
today but I should be able to get to it again tomorrow and try it.

Thanks for all your suggestions and comments!  When I have more info, I
will report back to this thread.  Hopefully we can solve it w/o a
reinstall.

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Michael George]

On Tue, Dec 06, 2005 at 08:03:55AM -0500, Michael George wrote:
> On Mon, Dec 05, 2005 at 05:29:19PM -0300, Arturo 'Buanzo' Busleiman wrote:
> > Michael George wrote:
> > > If anyone has any suggestions, I'd like to find a way out of this other
> > > than reinstalling the system.  Such a thing might happen again in the
> > > future to a non-development system and I'd like to know the way to get it
> > > working again.
> > 
> > Any kernel messages or syslog entries during the chrooted work?
> 
> Embarassingly, I didn't think to check that...  I am out with meetings
> today but I should be able to get to it again tomorrow and try it.
> 
> Thanks for all your suggestions and comments!  When I have more info, I
> will report back to this thread.  Hopefully we can solve it w/o a
> reinstall.

Okay, I have to oddities:
1. system will not proceed in boot process beyond the "configuring
system to use udev"
2. when chrooting into the system, some ports will not build.  Notably,
udev will not (re)build.

I booted the system (to the 2005.1 Live/Universal CD), chrooted, and I'm
trying to re-emerge udev.  Using debug mode on the emerge, I get:

livecd / # emerge -vd udev
Calculating dependencies
Parent:    None
Depstring: sys-fs/udev
Candidates: ['sys-fs/udev']
ebuild: sys-fs/udev-070-r1
binpkg: None
 -
Parent:    ebuild / sys-fs/udev-070-r1 merge
Depstring: sys-apps/hotplug-base !bootstrap? ( sys-devel/patch )
!bootstrap? ( sys-devel/patch ) sys-apps/hotplug-base
>=sys-apps/baselayout-1.8.6.12-r3
Exiting... None
 ...done!
>>> emerge (1 of 1) sys-fs/udev-070-r1 to /

/var/log/messages (outside the chroot) has nothing written to it to
indicate disk errors or anything.

dmesg also shows no new entries.

I can, though emerge other ports, like hotplug, with no problem.

Back to problem #1:

Another thing I noticed is that "route" inside the chroot will not
complete.  It prints the headers and then sits there doing nothing.

I checked the logs for the system on the HDD, and it appears that the
boot process cannot open /dev/console and agetty cannot open any of the
/dev/ttyX devices.

I am running baselayout 1.11.13-r1, so it's not supposed to have
problems if the files aren't there.  In addition /dev on that partition
is fully populated (though /dev/console has 600 perms, not 660)...

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Michael George]

Changing the perms of /dev/console to 660 didn't help much.  I do get
"Populating /dev/with device nodes ..." (which I don't think I got
before), but it still stops there.

Is the boot process trying to unload a tarball or something that it
might be choking on?  RC_DEVICE_TARBALL is set to "yes", but setting it
to "no" didn't help before...

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Michael George]

Actually, I spoke too soon.  The change in perms did get it further in
the boot process, but it complained about the special device /dev/sda1
(for /boot) not existing.  But it continued...

It seems to be stopped at "Caching service dependencies" now, but I will
check /var/log/messages and /var/log/dmesg and see what it tells me.

<time passes>

Looking at /var/log/messages, it appears that /dev/console and /dev/ttyX
were still not found, so /dev is not being correctly created, I don't
think.

It also says that the SCSI device was attached at /dev/sda, so the file
should have been there.

The last message in the log is from slapd, so it appears that the boot
process got quite a way through what it was supposed to.  However, it
never did give me a login prompt, just stopped at "Caching service
dependencies".  I think that's because /dev/ttyX couldn't be opened.  So
the system basically booted all it's services and had / mounted because
grub did that (didn't need /dev/sda3).  I wouldn't consider it a stable
system, though, w/o /dev being right.

Anyone know of a way to fix this issue?

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Richard Fish]

On 12/6/05, Michael George <george@mutualdata.com> wrote:
> Actually, I spoke too soon.  The change in perms did get it further in
> the boot process, but it complained about the special device /dev/sda1
> (for /boot) not existing.  But it continued...
>
> It seems to be stopped at "Caching service dependencies" now, but I will
> check /var/log/messages and /var/log/dmesg and see what it tells me.
>
> <time passes>
>
> Looking at /var/log/messages, it appears that /dev/console and /dev/ttyX
> were still not found, so /dev is not being correctly created, I don't
> think.
>
> It also says that the SCSI device was attached at /dev/sda, so the file
> should have been there.
>
> The last message in the log is from slapd, so it appears that the boot
> process got quite a way through what it was supposed to.  However, it
> never did give me a login prompt, just stopped at "Caching service
> dependencies".  I think that's because /dev/ttyX couldn't be opened.  So
> the system basically booted all it's services and had / mounted because
> grub did that (didn't need /dev/sda3).  I wouldn't consider it a stable
> system, though, w/o /dev being right.
>
> Anyone know of a way to fix this issue?

A couple of thoughts:

Make sure the /dev/null and /dev/console exist on the root filesystem
(boot from a livecd or "mount --bind / /mnt/root").

Also I recommend setting RC_DEVICE_TARBALL=no and RC_USE_FSTAB=no in
/etc/conf.d/rc

-Richard

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Michael George]

On Tue, Dec 06, 2005 at 01:06:39PM -0700, Richard Fish wrote:
> On 12/6/05, Michael George <george@mutualdata.com> wrote:
> > Actually, I spoke too soon.  The change in perms did get it further in
> > the boot process, but it complained about the special device /dev/sda1
> > (for /boot) not existing.  But it continued...
> >
> > It seems to be stopped at "Caching service dependencies" now, but I will
> > check /var/log/messages and /var/log/dmesg and see what it tells me.
> >
> > <time passes>
> >
> > Looking at /var/log/messages, it appears that /dev/console and /dev/ttyX
> > were still not found, so /dev is not being correctly created, I don't
> > think.
> >
> > It also says that the SCSI device was attached at /dev/sda, so the file
> > should have been there.
> >
> > The last message in the log is from slapd, so it appears that the boot
> > process got quite a way through what it was supposed to.  However, it
> > never did give me a login prompt, just stopped at "Caching service
> > dependencies".  I think that's because /dev/ttyX couldn't be opened.  So
> > the system basically booted all it's services and had / mounted because
> > grub did that (didn't need /dev/sda3).  I wouldn't consider it a stable
> > system, though, w/o /dev being right.
> >
> > Anyone know of a way to fix this issue?
> 
> A couple of thoughts:
> 
> Make sure the /dev/null and /dev/console exist on the root filesystem
> (boot from a livecd or "mount --bind / /mnt/root").

Already checked that.  The root filesystem does contain /dev/null and
/dev/console are there.  However, something is obviously breaking when
udev mounts the /dev system and then those devices are no longer
there...

> Also I recommend setting RC_DEVICE_TARBALL=no and RC_USE_FSTAB=no in
> /etc/conf.d/rc

I have tried RC_DEVICE_TARBALL=no and that didn't make a difference.  I
think RC_USE_FSTAB=no is the default.

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Richard Fish]

On 12/6/05, Michael George <george@mutualdata.com> wrote:
> Already checked that.  The root filesystem does contain /dev/null and
> /dev/console are there.  However, something is obviously breaking when
> udev mounts the /dev system and then those devices are no longer
> there...

Hmm, take a look at the rules files in /etc/udev/rules.d, and see if
there is any obvious corruption there.  Same thing with
/etc/udev/permissions.d/.

At this point, you might have to do something drastic:

rm -rvf /etc/udev && emerge --oneshot udev

-Richard

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Michael George]

On Tue, Dec 06, 2005 at 09:17:28PM -0700, Richard Fish wrote:
> On 12/6/05, Michael George <george@mutualdata.com> wrote:
> > Already checked that.  The root filesystem does contain /dev/null and
> > /dev/console are there.  However, something is obviously breaking when
> > udev mounts the /dev system and then those devices are no longer
> > there...
> 
> Hmm, take a look at the rules files in /etc/udev/rules.d, and see if
> there is any obvious corruption there.  Same thing with
> /etc/udev/permissions.d/.
> 
> At this point, you might have to do something drastic:
> 
> rm -rvf /etc/udev && emerge --oneshot udev

I'd like to do that, but the problem is that an emerge of udev cannot
complete.  Doesn't get started as a matter of fact.  I'm suspecting
corruption in the FS, but it's quite weird that I can emerge some things
just fine, btu others never get past the >>>>>>>>>>emerge message...

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Michael George]

On Wed, Dec 07, 2005 at 11:10:33AM -0500, Michael George wrote:
> On Tue, Dec 06, 2005 at 09:17:28PM -0700, Richard Fish wrote:
> > On 12/6/05, Michael George <george@mutualdata.com> wrote:
> > > Already checked that.  The root filesystem does contain /dev/null and
> > > /dev/console are there.  However, something is obviously breaking when
> > > udev mounts the /dev system and then those devices are no longer
> > > there...
> > 
> > Hmm, take a look at the rules files in /etc/udev/rules.d, and see if
> > there is any obvious corruption there.  Same thing with
> > /etc/udev/permissions.d/.
> > 
> > At this point, you might have to do something drastic:
> > 
> > rm -rvf /etc/udev && emerge --oneshot udev
> 
> I'd like to do that, but the problem is that an emerge of udev cannot
> complete.  Doesn't get started as a matter of fact.  I'm suspecting
> corruption in the FS, but it's quite weird that I can emerge some things
> just fine, btu others never get past the >>>>>>>>>>emerge message...

Well, nothing I do can address the problem.  The last things I've done:

I checked the consistency of the disks at the 3ware card, I fsck'd the
filesystems, I tried not using the tarball for the /dev directory, I
changed the permissions of /dev/console to be 660 as per the udev docs.
Nothing seemed to work.

I wanted to pull a tarball of the whole system first, though.  So I
booted from CD and chrooted to the HDD partitions.  Tar could not
complete.  If I didn't chroot, tar could complete just fine.  My only
conclusion is that updating the system with bad RAM caused some weird
problems in the filesystem which had nothing to do with consistency.
Even though the problems are VERY strange and I'm a bit skeptical about
this explanation, I can think of no other explanation.

We know the problem happened after an update.  We know that the RAM
failed at memtest86.  We know that the system cannot boot. We know that
the system cannot rebuild itself in a chroot.  We know that the system
couldn't even tarball itself in chroot.

We know that the tar, of the same data, could complete just fine using
the exectuable when booted from the CD-ROM.

I wish I had a way to recover, but I can think of none.  The CD isn't
equipped to do emerges (AFAIK) and emerge cannot be told to do a "virtual
chroot" (use the executables on the CD to emerge software in a chroot,
updating portate info inside that chroot -- I'm sure there are serious
issues with wanting to do this, like gcc and glibc versions, but it
would still have helped), I have no choice but reinstall the system and
start over with it.

I'm posting this so that the thread may have closure and whomever might
find it in the archives will know what I tried and that I found no
solution.

Thank you everyone for your helpful advice.

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[doug.marshall]

> About a week later, I booted it and it wouldn't boot.  It got stuck at
> "configuring system to use udev" and never gets okay.  I just sits there
> for ever.

My system hangs at this point quite often. I have no idea why - it seems to be quite random but if I try 
again I usually get it to boot. I take it your system never gets past this line on boot?
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Michael George]

On Mon, Dec 05, 2005 at 09:57:47PM +0100, doug.marshall@ias.fr wrote:
> > About a week later, I booted it and it wouldn't boot.  It got stuck at
> > "configuring system to use udev" and never gets okay.  I just sits there
> > for ever.
> 
> My system hangs at this point quite often. I have no idea why - it seems to be quite random but if I try 
> again I usually get it to boot. I take it your system never gets past this line on boot?

correct.  I've left it for minutes several times.  Never goes beyond...

-- 
-M

There are 10 kinds of people in this world:
	Those who can count in binary and those who cannot.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] system stopped booting

[Manuel McLure]

Michael George wrote:
> On Mon, Dec 05, 2005 at 09:57:47PM +0100, doug.marshall@ias.fr wrote:
> 
>>>About a week later, I booted it and it wouldn't boot.  It got stuck at
>>>"configuring system to use udev" and never gets okay.  I just sits there
>>>for ever.
>>
>>My system hangs at this point quite often. I have no idea why - it seems to be quite random but if I try 
>>again I usually get it to boot. I take it your system never gets past this line on boot?
> 
> 
> correct.  I've left it for minutes several times.  Never goes beyond...
> 

Take a look at the following link - it's a known bug in udev and LDAP. 
The workaround is listed in the bug notes.

http://bugs.gentoo.org/show_bug.cgi?id=99564

-- 
Manuel A. McLure KE6TAW <manuel@mclure.org> <http://www.mclure.org>
...for in Ulthar, according to an ancient and significant law,
no man may kill a cat.                       -- H.P. Lovecraft
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] gtk display

[cucu ionut cristian]

hi all !
I have a problem reguarding my gtk aplications they cannot be started
using sudo nmapfe(or any program for that matter)
the error message is Gtk-WARNING **: cannot open display:
i'm using the e17 windows manager, if it makes any differences
Thanks!

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] gtk display

[Holly Bostick]

cucu ionut cristian schreef:
> hi all !
> I have a problem reguarding my gtk aplications they cannot be started
> using sudo nmapfe(or any program for that matter)
> the error message is Gtk-WARNING **: cannot open display:
> i'm using the e17 windows manager, if it makes any differences
> Thanks!
> 

This occurs only with sudo?

I had a similar problem, solved by adding the following to visudoers:

# Uncomment to allow users in group wheel to export variables
Defaults:%wheel !env_reset

# Allow users in group users to export specific variables
# Defaults:%users       env_keep=TZ
==> Defaults:%users         env_keep=DISPLAY

There are probably other ways to solve this, but that works for me.

Hope it helps you.

Holly


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] gtk display

[cucu ionut cristian]

> This occurs only with sudo?
Yes
> I had a similar problem, solved by adding the following to visudoers:
> 
> # Uncomment to allow users in group wheel to export variables
> Defaults:%wheel !env_reset
> 
> # Allow users in group users to export specific variables
> # Defaults:%users       env_keep=TZ
> ==> Defaults:%users         env_keep=DISPLAY
> 
> There are probably other ways to solve this, but that works for me.
> 
> Hope it helps you.
It did solve my problem Thanks!
> Holly
> 
> 

-- 
gentoo-user@gentoo.org mailing list