[gentoo-user] Suspend-to-ram freezing tasks failed

[gentoo-user] Suspend-to-ram freezing tasks failed

[Серега Филатов]

Hello. I am upset about one issue that pops up from time to time which
I see now even more frequently than before. This has literally become
a pain in some place for me.
The issue is simple: gentoo, linux kernel 4.19.97, regular asus
laptop. I'm doing my business, closing the lid. I expect it to suspend
and stay in that mode.
And my laptop doesn't.

Some time ago the reason was mounted sshfs (I see fuse stuff in the
backtrace). Today, as you can see, baloo from KDE. Sometimes I see
"file.so".
When closing the lid it try for 20 seconds and gives up. Then some
time it tries again, always without success. Suddenly it gives up to
suspend and stays in a working state or some weird state when fan
still spinning but slowing down, power light is on and it wakes up
only by power button. It can discharge during the night or I can just
throw it in a bag where it stays awake and become hot.

Is there some way to tweak a kernel to not do that? I've noticed that
it is not only the gentoo or this laptop. I see this on different
hardware and different system (ubuntu specifically). I started to
think if it's a systemd thing because I've started to see it probably
the same time ubuntu switched to systemd.

Windows have no issues to sleep on the same hardware and suspends every time.

Part of dmesg with this message. I closed the lid, power light is on.
After a while I opened it and stroke a power button:

[  283.880396] PM: suspend entry (deep)
[  283.880398] PM: Syncing filesystems ... done.
[  284.176615] Freezing user space processes ...
[  304.175823] Freezing of tasks failed after 20.004 seconds (1 tasks
refusing to freeze, wq_busy=0):
[  304.175864] baloo_file_extr D13184 11112  11111 0x00000004
[  304.175874] Call Trace:
[  304.175888]  ? __schedule+0x26f/0x720
[  304.175893]  schedule+0x2d/0x80
[  304.175898]  io_schedule+0xd/0x30
[  304.175904]  generic_file_read_iter+0x319/0xa30
[  304.175910]  ? page_cache_tree_insert+0xd0/0xd0
[  304.175915]  __vfs_read+0x11e/0x170
[  304.175920]  vfs_read+0x98/0x150
[  304.175924]  ksys_pread64+0x60/0xa0
[  304.175929]  do_syscall_64+0x43/0xf0
[  304.175933]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  304.175937] RIP: 0033:0x7fae63191733
[  304.175947] Code: Bad RIP value.
[  304.175949] RSP: 002b:00007ffd5069d670 EFLAGS: 00000293 ORIG_RAX:
0000000000000011
[  304.175953] RAX: ffffffffffffffda RBX: 00005573b13dcff0 RCX: 00007fae63191733
[  304.175955] RDX: 0000000000000100 RSI: 00005573b13dd031 RDI: 0000000000000013
[  304.175956] RBP: 00005573b13dd031 R08: 0000000000000000 R09: 0000000000000007
[  304.175958] R10: 0000000002b08e9a R11: 0000000000000293 R12: 00007ffd5069dac8
[  304.175960] R13: 00007ffd5069d7d0 R14: 00000000ffffffff R15: 00007ffd5069d860
[  304.175987] OOM killer enabled.
[  304.175988] Restarting tasks ... done.
[  304.193089] PM: suspend exit
[  304.193150] PM: suspend entry (s2idle)
[  304.193151] PM: Syncing filesystems ... done.
[  304.626894] Freezing user space processes ... (elapsed 0.002 seconds) done.
[  304.629577] OOM killer disabled.
[  304.629578] Freezing remaining freezable tasks ... (elapsed 0.000
seconds) done.
[  304.630520] Suspending console(s) (use no_console_suspend to debug)
[  304.631663] wlp2s0: deauthenticating from xx:xx:xx:xx:xx:xx by
local choice (Reason: 3=DEAUTH_LEAVING)
[  304.640435] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[  304.640478] sd 2:0:0:0: [sdb] Synchronizing SCSI cache
[  304.640656] sd 0:0:0:0: [sda] Stopping disk
[  304.644025] sd 2:0:0:0: [sdb] Stopping disk
[  305.415243] ACPI: EC: interrupt blocked
[  751.841563] ACPI: EC: interrupt unblocked
[  751.983903] hpet1: lost 310 rtc interrupts
[  751.991105] sd 0:0:0:0: [sda] Starting disk
[  751.991126] sd 2:0:0:0: [sdb] Starting disk
[  751.993715] iwlwifi 0000:02:00.0: Applying debug destination EXTERNAL_DRAM
[  752.124938] iwlwifi 0000:02:00.0: Applying debug destination EXTERNAL_DRAM
[  752.192844] iwlwifi 0000:02:00.0: FW already configured (0) - re-configuring
[  752.302228] ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[  752.302619] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[  752.306480] ata3.00: configured for UDMA/133
[  754.031499] ata1.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
[  754.031504] ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE
LOCK) filtered out
[  754.031508] ata1.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE
CONFIGURATION OVERLAY) filtered out
[  754.035587] ata1.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES) succeeded
[  754.035592] ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY FREEZE
LOCK) filtered out
[  754.035596] ata1.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE
CONFIGURATION OVERLAY) filtered out
[  754.037029] ata1.00: configured for UDMA/100
[  754.045288] ata3.00: Enabling discard_zeroes_data
[  754.046007] OOM killer enabled.
[  754.046009] Restarting tasks ... done.
[  754.195279] PM: suspend exit

-- 
Sergey Filatov (raxp)
Telegram: @raxpyraxp

[gentoo-user] Initial console messages garbled

[Robin Atwood]

I posted about this on the Gentoo forums (see [1]) but didn't get any
response so I'll try my luck here! The post explains everything but.
briefly, normally when I boot a kernel I initially get some messages
in a large clunky font, and then the frame-buffer module loads and
everything is hi-res and fills the entire monitor. On my new box the
first messages are video-static until the frame-buffer takes over. I
have an error in a new root partition I have built and need to read
those messages! What of all the numerous parameters in various places
should I try tweaking?

Thanks
Robin

1. http://forums.gentoo.org/viewtopic-t-1114668-highlight-.html

-- 
----------------------------------------------------------------------
Robin Atwood.
----------------------------------------------------------------------

Re: [gentoo-user] Initial console messages garbled

[Jack]

On 2020.06.24 12:39, Robin Atwood wrote:
> I posted about this on the Gentoo forums (see [1]) but didn't get any  
> response so I'll try my luck here! The post explains everything but.  
> briefly, normally when I boot a kernel I initially get some messages  
> in a large clunky font, and then the frame-buffer module loads and  
> everything is hi-res and fills the entire monitor. On my new box the  
> first messages are video-static until the frame-buffer takes over. I  
> have an error in a new root partition I have built and need to read  
> those messages! What of all the numerous parameters in various places  
> should I try tweaking?
> 
> Thanks
> Robin
> 
> 1. http://forums.gentoo.org/viewtopic-t-1114668-highlight-.html

I can't answer your question, but I can suggest that you start a new  
thread with a new message, not replying to an old message, even if you  
do change the subject.  Many email readers thread discussions using  
internal message headers, not just subject, so it increases the chance  
your message may not be seen by someone who can answer.

Jack

Re: [gentoo-user] Initial console messages garbled

[Robin Atwood]

On Wed, 24 Jun 2020 12:54:32 -0400
Jack <ostroffjh@users.sourceforge.net> wrote:
> I can't answer your question, but I can suggest that you start a new  
> thread with a new message, not replying to an old message, even if
> you do change the subject.  Many email readers thread discussions
> using internal message headers, not just subject, so it increases the
> chance your message may not be seen by someone who can answer.
> 
> Jack

Thanks for pointing that out, it was the easiest way to post!

Robin
-- 

Re: [gentoo-user] Initial console messages garbled

[tedheadster]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

Robin,
 are you comfortable just going with a bare-bones console and build a new
kernel where you _disable_ CONFIG_FB? That might do it.

Alternately, you can hook up a serial cable to another computer and set
"console=ttyS0,115200n8".

- Matthew

[-- Attachment #2: Type: text/html, Size: 982 bytes --]

Re: [gentoo-user] Initial console messages garbled

[Robin Atwood]

On Wed, 24 Jun 2020 13:31:30 -0400
tedheadster <tedheadster@gmail.com> wrote:

> Robin,
>  are you comfortable just going with a bare-bones console and build a
> new kernel where you _disable_ CONFIG_FB? That might do it.
> 
> Alternately, you can hook up a serial cable to another computer and
> set "console=ttyS0,115200n8".

I will try that, if it works it will at least give me a chance to look
at the error messages.

I don't think I have any serial cables!

Thanks
Robin
-- 

Re: [gentoo-user] Initial console messages garbled

[Robin Atwood]

On Thu, 25 Jun 2020 13:55:18 +0700
> > Robin,
> >  are you comfortable just going with a bare-bones console and build
> > a new kernel where you _disable_ CONFIG_FB? That might do it.
> > 
> > Alternately, you can hook up a serial cable to another computer and
> > set "console=ttyS0,115200n8".  
> 
> I will try that, if it works it will at least give me a chance to look
> at the error messages.

I couldn't disable CONFIG_FB in the kernel, only select "module" or
"built-in" mode. I guess there is another parameter which needs
disabling. Researching that, I discovered you can disable frame-buffer
on the kernel command line. Also, that there might be an option in the
BIOS to set but I checked that and found nothing. So I rebooted and
used Grub to edit the kernel parameters and set "VGA=NORMAL NOMODESET".
This resulted in the usual initial garbage which was not cured by
loading the frame-buffer module. So disabling frame-buffer support made
things worse, which is rather what I was expecting. :(

Given the almost total lack of response to my post, I guess I am stuck
with this problem.

Thanks
Robin
-- 

Re: [gentoo-user] Initial console messages garbled

[Dale]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

Robin Atwood wrote:
>
> Given the almost total lack of response to my post, I guess I am stuck
> with this problem.
>
> Thanks
> Robin


The likely reason you are not getting many responses, you replied to a
message in another thread and only changed the subject line.  I don't
know anything about suspend to ram so I been marking this thread as read
as new replies appeared.  When I clicked on the folder for -user mailing
list, then I noticed the change in the subject since the thread was
expanded. I think this is the first message I saw on this subtopic.  I
suspect, others on this list are doing the same thing as me. 

What I would suggest is starting a new thread about your topic.  That
way people will be more likely to see your topic instead of the topic
that someone else started that is not related to this. 

Just a thought.  It may not help but you won't know until you try. 

Dale

:-)  :-) 

[-- Attachment #2: Type: text/html, Size: 1367 bytes --]

Re: [gentoo-user] Initial console messages garbled

[Rich Freeman]

On Thu, Jun 25, 2020 at 2:55 AM Robin Atwood <robin@binro.org> wrote:
>
> On Wed, 24 Jun 2020 13:31:30 -0400
> tedheadster <tedheadster@gmail.com> wrote:
>
> > Robin,
> >  are you comfortable just going with a bare-bones console and build a
> > new kernel where you _disable_ CONFIG_FB? That might do it.
> >
> > Alternately, you can hook up a serial cable to another computer and
> > set "console=ttyS0,115200n8".
>
> I will try that, if it works it will at least give me a chance to look
> at the error messages.
>
> I don't think I have any serial cables!
>

While serial consoles are one solution, I'd take a look at network
consoles.  They're FAR easier to manage on commodity hardware.  All
you really need is another host on the network that can run netcat.

I stick this in my /etc/grub/default - or otherwise get it onto the
command line in the bootloader:
netconsole=@/,6666@192.168.1.1

That tells the kernel to send all console output over UDP to
192.168.1.1:6666.  If you have multiple interfaces/etc you might need
to expand that command line a bit.  I have no idea how it comes up
with the sending IP - if you care about that you can specify it.  I'm
guessing it doesn't run DHCP - but this is just plain UDP so it is
one-way and there is no need for acks to get back to the sender.

On the destination host I run:
nc -u -l -p 6666

(nc is provided by the netcat package - a very basic tool that should
be available everywhere - probably on non-linux operating systems
also)

Start up the reception part before you try booting the host you're
troubleshooting, because it is just going to send packets blind into
the ether and if nothing is listening they're gone.  Obviously
netconsole is a very simple implementation so that it can run during
early boot.  It is very good for capturing panics/etc.

I don't know how it compares with serial console in terms of how early
it starts.  I think it does capture stuff very early in boot though -
both systems require a degree of hardware initialization before they
can work, but both are also very simple.

This does need to be enabled in the kernel.

You can also enable this on a running kernel but of course that does
no good for issues during boot.  Full docs are at:
https://www.kernel.org/doc/Documentation/networking/netconsole.txt


-- 
Rich

Re: [gentoo-user] Initial console messages garbled

[Sid Spry]

> On Thu, Jun 25, 2020 at 2:55 AM Robin Atwood <robin@binro.org> wrote:
> >
> > On Wed, 24 Jun 2020 13:31:30 -0400
> > tedheadster <tedheadster@gmail.com> wrote:
> >
> > > Robin,
> > >  are you comfortable just going with a bare-bones console and build a
> > > new kernel where you _disable_ CONFIG_FB? That might do it.
> > >
> > > Alternately, you can hook up a serial cable to another computer and
> > > set "console=ttyS0,115200n8".
> >
> > I will try that, if it works it will at least give me a chance to look
> > at the error messages.
> >
> > I don't think I have any serial cables!
> >
> 

Well, you may need to buy a cable one way or another if your issue is very early in the boot process, typically qualified as "before the console comes up." Netconsole may help you though.

Most motherboards used to have potentially unpopulated serial port headers on them. Those seen to be disappearing.

The replacement is https://www.kernel.org/doc/html/v5.4/driver-api/usb/usb3-debug-port.html. Most (all?) desktop xHCI controllers support a device mode that is essentially a very high speed CBC ACM serial port. This is especially useful for debugging laptops.

You do need to either make or buy the special A to A cable.

Re: [gentoo-user] Suspend-to-ram freezing tasks failed

[Michael]

[-- Attachment #1: Type: text/plain, Size: 6150 bytes --]

On Wednesday, 24 June 2020 16:30:25 BST Серега Филатов wrote:
> Hello. I am upset about one issue that pops up from time to time which
> I see now even more frequently than before. This has literally become
> a pain in some place for me.
> The issue is simple: gentoo, linux kernel 4.19.97, regular asus
> laptop. I'm doing my business, closing the lid. I expect it to suspend
> and stay in that mode.
> And my laptop doesn't.
> 
> Some time ago the reason was mounted sshfs (I see fuse stuff in the
> backtrace). Today, as you can see, baloo from KDE. Sometimes I see
> "file.so".

Ahh, yes, the baloo pain ...

You could configure baloo to only index parts of your filesystem you care to 
find when running semantic searches.  I typically switch it off completely, or 
only allow it to index a small part of the /home filesystem.  If you have it 
enabled and it indexes not only file names but also content, plus emails and 
what have you, then it will chew up an inordinate amount of resources.


> When closing the lid it try for 20 seconds and gives up. Then some
> time it tries again, always without success. Suddenly it gives up to
> suspend and stays in a working state or some weird state when fan
> still spinning but slowing down, power light is on and it wakes up
> only by power button. It can discharge during the night or I can just
> throw it in a bag where it stays awake and become hot.

Not good!  You could lose data like this.


> Is there some way to tweak a kernel to not do that? I've noticed that
> it is not only the gentoo or this laptop. I see this on different
> hardware and different system (ubuntu specifically). I started to
> think if it's a systemd thing because I've started to see it probably
> the same time ubuntu switched to systemd.
> 
> Windows have no issues to sleep on the same hardware and suspends every
> time.
> 
> Part of dmesg with this message. I closed the lid, power light is on.
> After a while I opened it and stroke a power button:
> 
> [  283.880396] PM: suspend entry (deep)
> [  283.880398] PM: Syncing filesystems ... done.
> [  284.176615] Freezing user space processes ...
> [  304.175823] Freezing of tasks failed after 20.004 seconds (1 tasks
> refusing to freeze, wq_busy=0):
> [  304.175864] baloo_file_extr D13184 11112  11111 0x00000004
> [  304.175874] Call Trace:
> [  304.175888]  ? __schedule+0x26f/0x720
> [  304.175893]  schedule+0x2d/0x80
> [  304.175898]  io_schedule+0xd/0x30
> [  304.175904]  generic_file_read_iter+0x319/0xa30
> [  304.175910]  ? page_cache_tree_insert+0xd0/0xd0
> [  304.175915]  __vfs_read+0x11e/0x170
> [  304.175920]  vfs_read+0x98/0x150
> [  304.175924]  ksys_pread64+0x60/0xa0
> [  304.175929]  do_syscall_64+0x43/0xf0
> [  304.175933]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [  304.175937] RIP: 0033:0x7fae63191733
> [  304.175947] Code: Bad RIP value.
> [  304.175949] RSP: 002b:00007ffd5069d670 EFLAGS: 00000293 ORIG_RAX:
> 0000000000000011
> [  304.175953] RAX: ffffffffffffffda RBX: 00005573b13dcff0 RCX:
> 00007fae63191733 [  304.175955] RDX: 0000000000000100 RSI: 00005573b13dd031
> RDI: 0000000000000013 [  304.175956] RBP: 00005573b13dd031 R08:
> 0000000000000000 R09: 0000000000000007 [  304.175958] R10: 0000000002b08e9a
> R11: 0000000000000293 R12: 00007ffd5069dac8 [  304.175960] R13:
> 00007ffd5069d7d0 R14: 00000000ffffffff R15: 00007ffd5069d860 [  304.175987]
> OOM killer enabled.
> [  304.175988] Restarting tasks ... done.
> [  304.193089] PM: suspend exit
> [  304.193150] PM: suspend entry (s2idle)
> [  304.193151] PM: Syncing filesystems ... done.
> [  304.626894] Freezing user space processes ... (elapsed 0.002 seconds)
> done. [  304.629577] OOM killer disabled.
> [  304.629578] Freezing remaining freezable tasks ... (elapsed 0.000
> seconds) done.
> [  304.630520] Suspending console(s) (use no_console_suspend to debug)
> [  304.631663] wlp2s0: deauthenticating from xx:xx:xx:xx:xx:xx by
> local choice (Reason: 3=DEAUTH_LEAVING)
> [  304.640435] sd 0:0:0:0: [sda] Synchronizing SCSI cache
> [  304.640478] sd 2:0:0:0: [sdb] Synchronizing SCSI cache
> [  304.640656] sd 0:0:0:0: [sda] Stopping disk
> [  304.644025] sd 2:0:0:0: [sdb] Stopping disk
> [  305.415243] ACPI: EC: interrupt blocked
> [  751.841563] ACPI: EC: interrupt unblocked
> [  751.983903] hpet1: lost 310 rtc interrupts
> [  751.991105] sd 0:0:0:0: [sda] Starting disk
> [  751.991126] sd 2:0:0:0: [sdb] Starting disk
> [  751.993715] iwlwifi 0000:02:00.0: Applying debug destination
> EXTERNAL_DRAM [  752.124938] iwlwifi 0000:02:00.0: Applying debug
> destination EXTERNAL_DRAM [  752.192844] iwlwifi 0000:02:00.0: FW already
> configured (0) - re-configuring [  752.302228] ata3: SATA link up 6.0 Gbps
> (SStatus 133 SControl 300) [  752.302619] ata1: SATA link up 6.0 Gbps
> (SStatus 133 SControl 300) [  752.306480] ata3.00: configured for UDMA/133
> [  754.031499] ata1.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES)
> succeeded [  754.031504] ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY
> FREEZE LOCK) filtered out
> [  754.031508] ata1.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE
> CONFIGURATION OVERLAY) filtered out
> [  754.035587] ata1.00: ACPI cmd ef/10:06:00:00:00:00 (SET FEATURES)
> succeeded [  754.035592] ata1.00: ACPI cmd f5/00:00:00:00:00:00 (SECURITY
> FREEZE LOCK) filtered out
> [  754.035596] ata1.00: ACPI cmd b1/c1:00:00:00:00:00 (DEVICE
> CONFIGURATION OVERLAY) filtered out
> [  754.037029] ata1.00: configured for UDMA/100
> [  754.045288] ata3.00: Enabling discard_zeroes_data
> [  754.046007] OOM killer enabled.
> [  754.046009] Restarting tasks ... done.
> [  754.195279] PM: suspend exit

You have OOM messages there.  It seems your baloo process eats up all your RAM 
and when the time comes to put the OS to sleep, there is not enough RAM to do 
so.  I don't know if using a swap file or swap partition would help, but 
controlling how much baloo is indexing will help both in terms of load on the 
CPU as well as in terms of RAM when awake and RAM when you suspend it. 

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Suspend-to-ram freezing tasks failed

[Серега Филатов]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

Hmm, isn't the "OOM killer enabled" just a notification that OOM killer is
working (not taking action)?

I'm running htop now. It seems that baloo really takes too much RAM and
CPU, but nothing fatal, plenty of free physical RAM and swap.

Anyway, thanks for the tip about baloo, but I think that it'll solve a
small part of the problem.

On Wed, Jun 24, 2020, 21:03 Michael <confabulate@kintzios.com> wrote:

> On Wednesday, 24 June 2020 16:30:25 BST Серега Филатов wrote:
> You have OOM messages there.  It seems your baloo process eats up all your
> RAM
> and when the time comes to put the OS to sleep, there is not enough RAM to
> do
> so.  I don't know if using a swap file or swap partition would help, but
> controlling how much baloo is indexing will help both in terms of load on
> the
> CPU as well as in terms of RAM when awake and RAM when you suspend it.

[-- Attachment #2: Type: text/html, Size: 1323 bytes --]

Re: [gentoo-user] Suspend-to-ram freezing tasks failed

[Michael]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On Wednesday, 24 June 2020 18:26:15 BST Серега Филатов wrote:
> Hmm, isn't the "OOM killer enabled" just a notification that OOM killer is
> working (not taking action)?
> 
> I'm running htop now. It seems that baloo really takes too much RAM and
> CPU, but nothing fatal, plenty of free physical RAM and swap.
> 
> Anyway, thanks for the tip about baloo, but I think that it'll solve a
> small part of the problem.

It should be easy to test.  In Plasma desktop go to SystemSettings > Search > 
File Search, then untick 'Enable File Search'.  Run 'balooctl disable' to make 
sure it is stopped, log out/in for good measure and then shut the lid, or 
select to Suspend to Ram from the menu while keeping an eye on the logs.

This has solved suspend problems on some of my systems, but not all.  There is 
still the odd desktop which misbehaves, randomly.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]