[gentoo-user] Problems booting vanilla kernel 4.1.x

[gentoo-user] Problems booting vanilla kernel 4.1.x

[Peter Weilbacher]

Dear all,

after successfully using kernel 4.0.5 (vanilla-sources) for a while, I
upgraded to 4.1.5 last week and 4.1.6 today. I cannot boot either of
them. On the screen I see

   Decompressing Linux... Parsing ELF... done.
   Booting the kernel.

as the last thing, then it just sits there.

To upgrade I copy the previously used .config to the new kernel
directory and run genkernel with --no-clean and --menuconfig so that I
get the same config as before -- unless I change something, which in
this case I didn't. (This has worked very nicely since 3.1.x or so).

Does that ring a bell with someone?
   Peter.

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Alexander Kapshuk]

On Sun, Aug 23, 2015 at 4:08 PM, Peter Weilbacher
<newsspam@weilbacher.org> wrote:
> Dear all,
>
> after successfully using kernel 4.0.5 (vanilla-sources) for a while, I
> upgraded to 4.1.5 last week and 4.1.6 today. I cannot boot either of
> them. On the screen I see
>
>    Decompressing Linux... Parsing ELF... done.
>    Booting the kernel.
>
> as the last thing, then it just sits there.
>
> To upgrade I copy the previously used .config to the new kernel
> directory and run genkernel with --no-clean and --menuconfig so that I
> get the same config as before -- unless I change something, which in
> this case I didn't. (This has worked very nicely since 3.1.x or so).
>
> Does that ring a bell with someone?
>    Peter.
>

I am running vanilla-sources 4.1.6, and so far I have not had any
trouble booting it.

Are you able to boot some of your previous kernels? If so, what does
your '/boot/grub/grub.cfg' look like?
What is the output of 'cat /etc/fstab' and 'ls -1 /boot'?

If you are not able to boot any of your kernels, if you could get a
hold of a Rescue CD or something like that and run the command lines
above, that would be helpful.

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Peter Weilbacher]

Hi Alexander,

On Sun, 23 Aug 2015, Alexander Kapshuk wrote:

> On Sun, Aug 23, 2015 at 4:08 PM, Peter Weilbacher <newsspam@weilbacher.org> wrote:
> >
> > after successfully using kernel 4.0.5 (vanilla-sources) for a while, I
> > upgraded to 4.1.5 last week and 4.1.6 today. I cannot boot either of
> > them. On the screen I see
> >
> >    Decompressing Linux... Parsing ELF... done.
> >    Booting the kernel.
> >
> > as the last thing, then it just sits there.
>
> I am running vanilla-sources 4.1.6, and so far I have not had any
> trouble booting it.
>
> Are you able to boot some of your previous kernels? If so, what does
> your '/boot/grub/grub.cfg' look like?
> What is the output of 'cat /etc/fstab' and 'ls -1 /boot'?

I can still boot 4.0.5 fine, with the same setup. I use lilo, and I
checked that I changed the two/four digits correctly in /etc/lilo.conf.

By chance I left the boot sit there for more than the typical minute,
and got multiple messages like

  INFO: rcu_sched self-detected stall on CPU { 3}  (t=60000 jiffies g=-256 c=-257 q=193)
  rcu_sched kthread starved for 50027 jiffies!

right after the above "Booting the kernel." line.

Do I need to activate a different kind of clocking or a CPU feature in
4.1.x?

   Peter.

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Alexander Kapshuk]

On Tue, Aug 25, 2015 at 7:40 PM, Peter Weilbacher
<newsspam@weilbacher.org> wrote:
> Hi Alexander,
>
> On Sun, 23 Aug 2015, Alexander Kapshuk wrote:
>
>> On Sun, Aug 23, 2015 at 4:08 PM, Peter Weilbacher <newsspam@weilbacher.org> wrote:
>> >
>> > after successfully using kernel 4.0.5 (vanilla-sources) for a while, I
>> > upgraded to 4.1.5 last week and 4.1.6 today. I cannot boot either of
>> > them. On the screen I see
>> >
>> >    Decompressing Linux... Parsing ELF... done.
>> >    Booting the kernel.
>> >
>> > as the last thing, then it just sits there.
>>
>> I am running vanilla-sources 4.1.6, and so far I have not had any
>> trouble booting it.
>>
>> Are you able to boot some of your previous kernels? If so, what does
>> your '/boot/grub/grub.cfg' look like?
>> What is the output of 'cat /etc/fstab' and 'ls -1 /boot'?
>
> I can still boot 4.0.5 fine, with the same setup. I use lilo, and I
> checked that I changed the two/four digits correctly in /etc/lilo.conf.
>
> By chance I left the boot sit there for more than the typical minute,
> and got multiple messages like
>
>   INFO: rcu_sched self-detected stall on CPU { 3}  (t=60000 jiffies g=-256 c=-257 q=193)
>   rcu_sched kthread starved for 50027 jiffies!
>
> right after the above "Booting the kernel." line.
>
> Do I need to activate a different kind of clocking or a CPU feature in
> 4.1.x?
>
>    Peter.
>

I've never experienced this particular kernel trouble myself, so I'm
not sure if my input would be of much help.
Here's what the kernel documentation has to say about this kind of issue:

/usr/src/linux/Documentation/RCU/stallwarn.txt:29,33
CONFIG_RCU_CPU_STALL_INFO

    This kernel configuration parameter causes the stall warning to
    print out additional per-CPU diagnostic information, including
    information on scheduling-clock ticks and RCU's idle-CPU tracking.

/usr/src/linux/Documentation/RCU/stallwarn.txt:104,109
If the CONFIG_RCU_CPU_STALL_INFO kernel configuration parameter is set,
more information is printed with the stall-warning message, for example:

    INFO: rcu_preempt detected stall on CPU
    0: (63959 ticks this GP) idle=241/3fffffffffffffff/0 softirq=82/543
       (t=65000 jiffies)

/usr/src/linux/Documentation/RCU/stallwarn.txt:240,250
To diagnose the cause of the stall, inspect the stack traces.
The offending function will usually be near the top of the stack.
If you have a series of stall warnings from a single extended stall,
comparing the stack traces can often help determine where the stall
is occurring, which will usually be in the function nearest the top of
that portion of the stack which remains the same from trace to trace.
If you can reliably trigger the stall, ftrace can be quite helpful.

RCU bugs can often be debugged with the help of CONFIG_RCU_TRACE
and with RCU's event tracing.  For information on RCU's event tracing,
see include/trace/events/rcu.h.

Have a look for possibly stack traces in these log files:
/var/log/{messages,dmesg}.

Hopefully, someone else with more kernel debugging experience will
have something more substantial to say about this.

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Peter Weilbacher]

On Tue, 25 Aug 2015, Alexander Kapshuk wrote:

> I've never experienced this particular kernel trouble myself, so I'm
> not sure if my input would be of much help.
> Here's what the kernel documentation has to say about this kind of issue:
>
> /usr/src/linux/Documentation/RCU/stallwarn.txt:29,33
[...]
> Have a look for possibly stack traces in these log files:
> /var/log/{messages,dmesg}.
>
> Hopefully, someone else with more kernel debugging experience will
> have something more substantial to say about this.

Thanks for the reply, Alex. I had googled for those messages, too, and
had found that RCU info. Unfortunately, I have no idea what to do with
it, since it stops right at the booting stage, right before init starts,
so before I can do anything interactive.

I have posted a message to the LKML, maybe someone there has a hint.
(The machine is old and rarely used, but I would still like to run the
newest software because it is connected to the net.)

   Peter.

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Alexander Kapshuk]

On Sat, Aug 29, 2015 at 3:09 PM, Peter Weilbacher
<newsspam@weilbacher.org> wrote:
> On Tue, 25 Aug 2015, Alexander Kapshuk wrote:
>
>> I've never experienced this particular kernel trouble myself, so I'm
>> not sure if my input would be of much help.
>> Here's what the kernel documentation has to say about this kind of issue:
>>
>> /usr/src/linux/Documentation/RCU/stallwarn.txt:29,33
> [...]
>> Have a look for possibly stack traces in these log files:
>> /var/log/{messages,dmesg}.
>>
>> Hopefully, someone else with more kernel debugging experience will
>> have something more substantial to say about this.
>
> Thanks for the reply, Alex. I had googled for those messages, too, and
> had found that RCU info. Unfortunately, I have no idea what to do with
> it, since it stops right at the booting stage, right before init starts,
> so before I can do anything interactive.
>
> I have posted a message to the LKML, maybe someone there has a hint.
> (The machine is old and rarely used, but I would still like to run the
> newest software because it is connected to the net.)
>
>    Peter.
>

Hope you do get this sorted.

You probably did look into this yourself, but did you double-check
your /etc/lilo.conf? Is everything fine there?

Here are my RCU kernel config options. What do yours look like?
% uname -r
4.1.6-vanilla
% grep RCU .config
# RCU Subsystem
CONFIG_TREE_RCU=y
CONFIG_SRCU=y
# CONFIG_TASKS_RCU is not set
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_FANOUT=32
CONFIG_RCU_FANOUT_LEAF=16
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_RCU_FAST_NO_HZ is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_RCU_KTHREAD_PRIO=0
# CONFIG_RCU_NOCB_CPU is not set
# CONFIG_RCU_EXPEDITE_BOOT is not set
# RCU Debugging
# CONFIG_PROVE_RCU is not set
# CONFIG_SPARSE_RCU_POINTER is not set
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_TIMEOUT=21
# CONFIG_RCU_CPU_STALL_INFO is not set
# CONFIG_RCU_TRACE is not set

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Peter Weilbacher]

On Sat, 29 Aug 2015, Alexander Kapshuk wrote:

> You probably did look into this yourself, but did you double-check
> your /etc/lilo.conf? Is everything fine there?

At least it's identical between 4.0.5 and 4.1.6:

image=/boot/kernel-genkernel-x86_64-4.0.5
   label=Linux405
   read-only # read-only for checking
   root=/dev/ram0
   append="init=/linuxrc keymap=de ramdisk=8192 real_root=/dev/sda6
           splash=verbose,theme:default console=tty1 quiet
           radeon.modeset=1 video=radeon:mtrr:3,ywrap,1680x1050-32@60
           ahci.marvell_enable=0"
   initrd=/boot/initramfs-genkernel-x86_64-4.0.5

Don't ask me where all those options came from, they grew with time...

> Here are my RCU kernel config options. What do yours look like?
> % uname -r
> 4.1.6-vanilla
> % grep RCU .config
> # RCU Subsystem
> CONFIG_TREE_RCU=y
> CONFIG_SRCU=y
> # CONFIG_TASKS_RCU is not set
> CONFIG_RCU_STALL_COMMON=y
> CONFIG_RCU_FANOUT=32
> CONFIG_RCU_FANOUT_LEAF=16
> # CONFIG_RCU_FANOUT_EXACT is not set
> # CONFIG_RCU_FAST_NO_HZ is not set
> # CONFIG_TREE_RCU_TRACE is not set
> CONFIG_RCU_KTHREAD_PRIO=0
> # CONFIG_RCU_NOCB_CPU is not set
> # CONFIG_RCU_EXPEDITE_BOOT is not set
> # RCU Debugging
> # CONFIG_PROVE_RCU is not set
> # CONFIG_SPARSE_RCU_POINTER is not set
> # CONFIG_RCU_TORTURE_TEST is not set
> CONFIG_RCU_CPU_STALL_TIMEOUT=21
> # CONFIG_RCU_CPU_STALL_INFO is not set
> # CONFIG_RCU_TRACE is not set

Thanks. The only difference to my config there is that I have
CONFIG_RCU_FANOUT=64 and CONFIG_RCU_CPU_STALL_TIMEOUT=60 (don't ask me
where the values come from, I don't remember ever setting them different
from the default). But since that is all the same between 4.0.5 and
4.1.6, I don't think it has anything to do with my problem. Could the
RCU message just be telling me that since the machine doesn't properly
boot, it doesn't have anything to do?

Cheers,
   Peter.

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Fernando Rodriguez]

On Tuesday, August 25, 2015 6:40:01 PM Peter Weilbacher wrote:
> Hi Alexander,
> 
> On Sun, 23 Aug 2015, Alexander Kapshuk wrote:
> 
> > On Sun, Aug 23, 2015 at 4:08 PM, Peter Weilbacher 
<newsspam@weilbacher.org> wrote:
> > >
> > > after successfully using kernel 4.0.5 (vanilla-sources) for a while, I
> > > upgraded to 4.1.5 last week and 4.1.6 today. I cannot boot either of
> > > them. On the screen I see
> > >
> > >    Decompressing Linux... Parsing ELF... done.
> > >    Booting the kernel.
> > >
> > > as the last thing, then it just sits there.
> >
> > I am running vanilla-sources 4.1.6, and so far I have not had any
> > trouble booting it.
> >
> > Are you able to boot some of your previous kernels? If so, what does
> > your '/boot/grub/grub.cfg' look like?
> > What is the output of 'cat /etc/fstab' and 'ls -1 /boot'?
> 
> I can still boot 4.0.5 fine, with the same setup. I use lilo, and I
> checked that I changed the two/four digits correctly in /etc/lilo.conf.
> 
> By chance I left the boot sit there for more than the typical minute,
> and got multiple messages like
> 
>   INFO: rcu_sched self-detected stall on CPU { 3}  (t=60000 jiffies g=-256 
c=-257 q=193)
>   rcu_sched kthread starved for 50027 jiffies!
> 
> right after the above "Booting the kernel." line.
> 
> Do I need to activate a different kind of clocking or a CPU feature in
> 4.1.x?
> 
>    Peter.
> 

Here's how I would go about it:

1. Add loglevel=7 to your kernel parameters and see what it prints before it 
hangs. 

2. Change your scheduler settings (ie. if you're using the preemptive 
scheduler or voluntary premption scheduler switch to the regular one) and try 
again.

3. From your kernel parameters I assume you're using the radeon free driver 
right? If that's the case disable it (don't compile it in or just delete the 
module) and try to boot wiith a framebuffer. If you're using the proprietary 
driver it has problem with preemptive kernels, with the 3.18.x series it 
started logging a lot of errors which I assume where warnings of some change 
yet to come.

4. If all else fails clone the kernel repo (be prepared to download a 2GB 
repo) and do a git bisect (google it) between the last kernel that worked and 
the first that didn't. That will eventually give you the exact commit that 
broke it. From there you can post on the mailing list for the relevant 
subsystem or you could try emailing the dev that commited it.

-- 
Fernando Rodriguez

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Peter Weilbacher]

Hi Fernando,

On Sun, 30 Aug 2015, Fernando Rodriguez wrote:

> 1. Add loglevel=7 to your kernel parameters and see what it prints before it
> hangs.

That helped, it showed me something about drm, so...

> 3. From your kernel parameters I assume you're using the radeon free driver
> right? If that's the case disable it (don't compile it in or just delete the
> module) and try to boot wiith a framebuffer.

... this was a good suggestion. Switching off DRM/Radeon gets me a
kernel that boots. However, with that config I cannot run X (which then
complains about missing kernel mode switching). If I follow
wiki.gentoo.org/wiki/Xorg/Guide#Kernel_modesetting I again arrive at the
kernel settings that I previously had and which didn't work with 4.1.x.

Any more suggestions?
   Peter.

[gentoo-user] Re: Problems booting vanilla kernel 4.1.x

[James]

Peter Weilbacher <newsspam <at> Weilbacher.org> writes:


> > 1. Add loglevel=7 to your kernel parameters and see what it prints 
> > before it hangs.

> Any more suggestions?
>    Peter.

Hello Peter,

Here is a great boot debugging resource that may help [1] 
It's an Arch doc, but there is plenty of information therein
that is generic in nature. Also there is the gentoo crash dump
doc that may help [2], but it is for system that are already running;
included in case you get that far and still have problems.

[1] https://wiki.archlinux.org/index.php/Boot_debugging#netconsole

[2] https://wiki.gentoo.org/wiki/Kernel_Crash_Dumps


hth,
James

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Fernando Rodriguez]

On Sunday, August 30, 2015 10:51:57 PM Peter Weilbacher wrote:
> Hi Fernando,
> 
> On Sun, 30 Aug 2015, Fernando Rodriguez wrote:
> 
> > 1. Add loglevel=7 to your kernel parameters and see what it prints before 
it
> > hangs.
> 
> That helped, it showed me something about drm, so...
> 
> > 3. From your kernel parameters I assume you're using the radeon free 
driver
> > right? If that's the case disable it (don't compile it in or just delete 
the
> > module) and try to boot wiith a framebuffer.
> 
> ... this was a good suggestion. Switching off DRM/Radeon gets me a
> kernel that boots. However, with that config I cannot run X (which then
> complains about missing kernel mode switching). If I follow
> wiki.gentoo.org/wiki/Xorg/Guide#Kernel_modesetting I again arrive at the
> kernel settings that I previously had and which didn't work with 4.1.x.
> 
> Any more suggestions?
>    Peter.
> 

At least you norrowed down, that was the idea.
I would suspect a new bug, so post to the radeon mailing list. Doing the git 
bisect first will make it easier for them so they'll be more willing to help. I 
would try booting without those radeon paremeters first.

You could try the proprietary driver but the one in the portage tree will not 
build with a kernel >3.18.19 but if you search b.g.o there are patches to make 
it build. Or you could try my ebuild but I'm not sure that it will build 
either since I'm using 3.18.20 now:

https://github.com/fernando-rodriguez/portage-overlay/tree/master/x11-drivers/ati-drivers


-- 
Fernando Rodriguez

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Fernando Rodriguez]

On Sunday, August 30, 2015 6:11:00 PM Fernando Rodriguez wrote:
> On Sunday, August 30, 2015 10:51:57 PM Peter Weilbacher wrote:
> > Hi Fernando,
> > 
> > On Sun, 30 Aug 2015, Fernando Rodriguez wrote:
> > 
> > > 1. Add loglevel=7 to your kernel parameters and see what it prints 
before 
> it
> > > hangs.
> > 
> > That helped, it showed me something about drm, so...
> > 
> > > 3. From your kernel parameters I assume you're using the radeon free 
> driver
> > > right? If that's the case disable it (don't compile it in or just delete 
> the
> > > module) and try to boot wiith a framebuffer.
> > 
> > ... this was a good suggestion. Switching off DRM/Radeon gets me a
> > kernel that boots. However, with that config I cannot run X (which then
> > complains about missing kernel mode switching). If I follow
> > wiki.gentoo.org/wiki/Xorg/Guide#Kernel_modesetting I again arrive at the
> > kernel settings that I previously had and which didn't work with 4.1.x.
> > 
> > Any more suggestions?
> >    Peter.
> > 
> 
> At least you norrowed down, that was the idea.
> I would suspect a new bug, so post to the radeon mailing list. Doing the git 
> bisect first will make it easier for them so they'll be more willing to help. 
I 
> would try booting without those radeon paremeters first.
> 
> You could try the proprietary driver but the one in the portage tree will 
not 
> build with a kernel >3.18.19 but if you search b.g.o there are patches to 
make 
> it build. Or you could try my ebuild but I'm not sure that it will build 
> either since I'm using 3.18.20 now:
> 
> https://github.com/fernando-rodriguez/portage-overlay/tree/master/x11-drivers/ati-drivers

And you should still try suggestion #2 because it's very likely to only affect 
one specific configuration.

-- 
Fernando Rodriguez

Re: [gentoo-user] Problems booting vanilla kernel 4.1.x

[Peter Weilbacher]

On Sun, 30 Aug 2015, Fernando Rodriguez wrote:

> And you should still try suggestion #2 because it's very likely to only affect
> one specific configuration.

Thanks, I did that just now. Took another, but didn't give me any more
data to go by. Getting set up for bisecting and doing real work on this
would probably fill my evenings until end of next week.

Nowadays I just don't have the time to fiddle around with computers any
more (and it's not fun any more, either). I guess, I'll just go back to
a kernel version with longterm support.

Thanks anyway for your help!
   Peter.