[gentoo-user] Debug memory leaks in X server

[gentoo-user] Debug memory leaks in X server

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 266 bytes --]

Hi list!

I have a suspicion that viewing certain PDFs in okular causes X server
to leak memory. Currently it is using 1.8 GB after 3 days uptime. Has
anyone else observed that? Is there a way to inspect X server's memory
usage?

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] Debug memory leaks in X server

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 2040 bytes --]

On Monday 26 Nov 2012 09:56:27 Florian Philipp wrote:
> Hi list!
> 
> I have a suspicion that viewing certain PDFs in okular causes X server
> to leak memory. Currently it is using 1.8 GB after 3 days uptime. Has
> anyone else observed that? Is there a way to inspect X server's memory
> usage?

I have noticed that okular recently started breaking into a sweat when it 
renders pdf files.  Even a four page document with a bit of colour and 
graphics seems to push the cpu:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16387 michael   23   3  466m  52m  30m S  102  1.3   0:36.88 /usr/bin/okular


In case someone can interpret the content of the file:
=====================================================
$ mupdfinfo Downloads/E016378510.pdf 
Downloads/E016378510.pdf:

PDF-1.4
Info object (18 0 R):
<<
  /CreationDate (D:20121126030121-05'00')
  /Subject <>
  /Author <>
  /Creator (PlanetPress Watch)
  /Keywords <>
  /Producer (Distiller)
  /ModDate (D:20121126030123-05'00')
  /Title (CONFIRMATIONS)
>>

Pages: 4

Retrieving info from pages 1-4...
Mediaboxes (1):
	    1 (     21 0 R): [ 0 0 612 792 ]

Fonts (2):
	    1 (     21 0 R): TrueType 'FIBAKI+GillSans' (23 0 R)
	    1 (     21 0 R): Type1 'GillSans-Bold' (28 0 R)

Images (7):
	    1 (     21 0 R): [ DCT ] 1125x1050 8bpc DevGray (31 0 R)
	    1 (     21 0 R): [ DCT ] 2550x3300 8bpc ICC (32 0 R)
	    1 (     21 0 R): [ Flate ] 481x164 8bpc ICC (33 0 R)
	    2 (      1 0 R): [ DCT ] 2550x3300 8bpc ICC (4 0 R)
	    2 (      1 0 R): [ DCT ] 1125x1050 8bpc DevGray (5 0 R)
	    3 (      6 0 R): [ DCT ] 2550x3300 8bpc ICC (9 0 R)
	    4 (     10 0 R): [ DCT ] 2549x3300 8bpc ICC (13 0 R)
=====================================================

Once rendered, the CPU goes down to normal levels.  The problem does not seem 
to occur when the pdf is just text, i.e. no photographs, or complex graphics 
in it.

Other than the various top apps, perhaps you can try lsof?
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Debug memory leaks in X server

[Dale]

Mick wrote:
> On Monday 26 Nov 2012 09:56:27 Florian Philipp wrote:
>> Hi list!
>>
>> I have a suspicion that viewing certain PDFs in okular causes X server
>> to leak memory. Currently it is using 1.8 GB after 3 days uptime. Has
>> anyone else observed that? Is there a way to inspect X server's memory
>> usage?
> I have noticed that okular recently started breaking into a sweat when it 
> renders pdf files.  Even a four page document with a bit of colour and 
> graphics seems to push the cpu:
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 16387 michael   23   3  466m  52m  30m S  102  1.3   0:36.88 /usr/bin/okular
> <<< SNIP >>>
>
> Once rendered, the CPU goes down to normal levels.  The problem does not seem 
> to occur when the pdf is just text, i.e. no photographs, or complex graphics 
> in it.
>
> Other than the various top apps, perhaps you can try lsof?

I have a local grocery store that I have to go to the website to get
their sale ads.  Anyway, it is generally 2 pages and even on this 4 core
rig with more than plenty of ram, it takes a bit to open them.  Funny
thing is, I have some that is about sewing, lots of pictures in those
since I need pics to get the idea, anyway, they load up in a flash.  As
soon as Okular loads, the pages are there. 

Since this is more like what Florian describes, I guess we see the same
things.  I'm not sure about ram itself but some files do open
differently.  By the way, the grocery ad is a much smaller file than the
sewing files.  Both in file size and number of pages.  One would expect
it to be the opposite. 

Looks like I have a problem that I didn't know I had.  With 16Gbs of
ram, I hadn't noticed anything with the ram, other than Seamonkey being
its usual hoggy self.  :/   I guess this is to sort of confirm that
someone else sees a similar thing to Florian. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!

Re: [gentoo-user] Debug memory leaks in X server

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 2843 bytes --]

On Monday 26 Nov 2012 12:43:38 Dale wrote:
> Mick wrote:
> > On Monday 26 Nov 2012 09:56:27 Florian Philipp wrote:
> >> Hi list!
> >> 
> >> I have a suspicion that viewing certain PDFs in okular causes X server
> >> to leak memory. Currently it is using 1.8 GB after 3 days uptime. Has
> >> anyone else observed that? Is there a way to inspect X server's memory
> >> usage?
> > 
> > I have noticed that okular recently started breaking into a sweat when it
> > renders pdf files.  Even a four page document with a bit of colour and
> > 
> > graphics seems to push the cpu:
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> > 
> > 16387 michael   23   3  466m  52m  30m S  102  1.3   0:36.88
> > /usr/bin/okular <<< SNIP >>>
> > 
> > Once rendered, the CPU goes down to normal levels.  The problem does not
> > seem to occur when the pdf is just text, i.e. no photographs, or complex
> > graphics in it.
> > 
> > Other than the various top apps, perhaps you can try lsof?
> 
> I have a local grocery store that I have to go to the website to get
> their sale ads.  Anyway, it is generally 2 pages and even on this 4 core
> rig with more than plenty of ram, it takes a bit to open them.  Funny
> thing is, I have some that is about sewing, lots of pictures in those
> since I need pics to get the idea, anyway, they load up in a flash.  As
> soon as Okular loads, the pages are there.
> 
> Since this is more like what Florian describes, I guess we see the same
> things.  I'm not sure about ram itself but some files do open
> differently.  By the way, the grocery ad is a much smaller file than the
> sewing files.  Both in file size and number of pages.  One would expect
> it to be the opposite.
> 
> Looks like I have a problem that I didn't know I had.  With 16Gbs of
> ram, I hadn't noticed anything with the ram, other than Seamonkey being
> its usual hoggy self.  :/   I guess this is to sort of confirm that
> someone else sees a similar thing to Florian.

This is not a RAM issue, but seemingly a CPU issue.  Furthermore, it does not 
seem to be related exclusively to okular.  I just tried qpdfview and it also 
took ages to open/render - HOWEVER - when I tried mupdf it was rendered in 
milliseconds and the CPU usage stayed very very low.  This may be something to 
do with the wonderful KDE and friends.

I recently upgraded to KDE-4.9.3 and this is not the only thing I noticed. In 
Kmail-1.13.7 all sent messages are saved in the local/sent-mail directory, 
irrespective of the path I enter in Kmail's settings for each email account.  
Initially I though that sent messages were being lost - not sent - but then I 
noticed that the default sent folder started getting larger.

I better start another thread for this problem.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Debug memory leaks in X server

[Dale]

Mick wrote:
> On Monday 26 Nov 2012 12:43:38 Dale wrote:
>> Mick wrote:
>>> On Monday 26 Nov 2012 09:56:27 Florian Philipp wrote:
>>>> Hi list!
>>>>
>>>> I have a suspicion that viewing certain PDFs in okular causes X server
>>>> to leak memory. Currently it is using 1.8 GB after 3 days uptime. Has
>>>> anyone else observed that? Is there a way to inspect X server's memory
>>>> usage?
>>> I have noticed that okular recently started breaking into a sweat when it
>>> renders pdf files.  Even a four page document with a bit of colour and
>>>
>>> graphics seems to push the cpu:
>>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>
>>> 16387 michael   23   3  466m  52m  30m S  102  1.3   0:36.88
>>> /usr/bin/okular <<< SNIP >>>
>>>
>>> Once rendered, the CPU goes down to normal levels.  The problem does not
>>> seem to occur when the pdf is just text, i.e. no photographs, or complex
>>> graphics in it.
>>>
>>> Other than the various top apps, perhaps you can try lsof?
>> I have a local grocery store that I have to go to the website to get
>> their sale ads.  Anyway, it is generally 2 pages and even on this 4 core
>> rig with more than plenty of ram, it takes a bit to open them.  Funny
>> thing is, I have some that is about sewing, lots of pictures in those
>> since I need pics to get the idea, anyway, they load up in a flash.  As
>> soon as Okular loads, the pages are there.
>>
>> Since this is more like what Florian describes, I guess we see the same
>> things.  I'm not sure about ram itself but some files do open
>> differently.  By the way, the grocery ad is a much smaller file than the
>> sewing files.  Both in file size and number of pages.  One would expect
>> it to be the opposite.
>>
>> Looks like I have a problem that I didn't know I had.  With 16Gbs of
>> ram, I hadn't noticed anything with the ram, other than Seamonkey being
>> its usual hoggy self.  :/   I guess this is to sort of confirm that
>> someone else sees a similar thing to Florian.
> This is not a RAM issue, but seemingly a CPU issue.  Furthermore, it does not 
> seem to be related exclusively to okular.  I just tried qpdfview and it also 
> took ages to open/render - HOWEVER - when I tried mupdf it was rendered in 
> milliseconds and the CPU usage stayed very very low.  This may be something to 
> do with the wonderful KDE and friends.
>
> I recently upgraded to KDE-4.9.3 and this is not the only thing I noticed. In 
> Kmail-1.13.7 all sent messages are saved in the local/sent-mail directory, 
> irrespective of the path I enter in Kmail's settings for each email account.  
> Initially I though that sent messages were being lost - not sent - but then I 
> noticed that the default sent folder started getting larger.
>
> I better start another thread for this problem.

That was mostly my point but I seemed to have glossed over the CPU part
after rereading my post.  My point was, ram is not a issue because I
have lots of it.  The subject mentions memory so I mostly went to it
instead of the CPU.  Even with that much ram, it takes a long time to
load, at least for some files.  It seems to be a CPU thing, not a memory
thing.  Maybe Florian knows something we don't know at this point. 

I don't know the exact percentages but I usually max out at least one
core when I open the grocery ad.  The sewing files, although larger,
open faster.  I just wonder what is used to make one as opposed to the
other?  Maybe that is the difference. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how you interpreted my words!

Re: [gentoo-user] Debug memory leaks in X server

[Volker Armin Hemmann]

Am Montag, 26. November 2012, 10:22:17 schrieb Mick:
> On Monday 26 Nov 2012 09:56:27 Florian Philipp wrote:
> > Hi list!
> > 
> > I have a suspicion that viewing certain PDFs in okular causes X server
> > to leak memory. Currently it is using 1.8 GB after 3 days uptime. Has
> > anyone else observed that? Is there a way to inspect X server's memory
> > usage?
> 
> I have noticed that okular recently started breaking into a sweat when it
> renders pdf files.  Even a four page document with a bit of colour and
> graphics seems to push the cpu:

not here?

hm.. maybe a poppler issue?

-- 
#163933

Re: [gentoo-user] Debug memory leaks in X server

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1439 bytes --]

Am 26.11.2012 10:56, schrieb Florian Philipp:
> Hi list!
> 
> I have a suspicion that viewing certain PDFs in okular causes X server
> to leak memory. Currently it is using 1.8 GB after 3 days uptime. Has
> anyone else observed that? Is there a way to inspect X server's memory
> usage?
> 
> Regards,
> Florian Philipp

Okay, I have now a better understanding of my memory issue (no clue
about Mick's CPU issue. Maybe you can post a link to a PDF to demonstrate?):

As it seems, Okular is abusing the X server to share pixmaps between
instances [1]. In theory, this reduces memory consumption and CPU time,
however, there are two issues:
a) Sometimes Okular is leaking allocations.
b) X cannot handle this usage pattern, sometimes leaks memory or cannot
deallocate it because of memory fragmentation [2].

Neither side seems to be willing to fix the issue. For now, my
workarounds will be:
- Playing around with Okular's memory settings to make the issue less
critical.
- Using frontswap and normal swap + high swappiness settings to swap out
the leaked memory.
- Looking for another PDF/djvu/ps viewer.

Maybe Wayland can handle the situation better in a few years.

[1] https://bugs.kde.org/show_bug.cgi?id=177213
[2] https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/98783

BTW: Regarding my question for an X memory inspector: x11-misc/xrestop
is the answer.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]