[gentoo-user] xz memory hungry?

[gentoo-user] xz memory hungry?

[Jorge Almeida]

# tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
xz: (stdin): Cannot allocate memory
tar: Child returned status 1
tar: Error is not recoverable: exiting now

The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
Emerging fails on m4. tar xJvf fails both from within the chroot and from the
host system. top shows that nothing is using any amount of memory worth
mentioning. Extracting libtool-2.4.tar.xz works. I can extract
m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
related, except that I have no choice, as m4 is pulled by other packages.
What to do?

app-arch/xz-utils-5.0.3 in chroot
xz 5.0.4 in host system (Archlinux)

TIA

Jorge Almeida

Re: [gentoo-user] xz memory hungry?

[Michael Mol]

On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
> xz: (stdin): Cannot allocate memory
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
> Emerging fails on m4. tar xJvf fails both from within the chroot and from the
> host system. top shows that nothing is using any amount of memory worth
> mentioning. Extracting libtool-2.4.tar.xz works. I can extract
> m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
> related, except that I have no choice, as m4 is pulled by other packages.
> What to do?
>
> app-arch/xz-utils-5.0.3 in chroot
> xz 5.0.4 in host system (Archlinux)

How much do you have free? From xz's manpage:

 Memory usage
       The memory usage of xz varies from a  few  hundred  kilobytes
to  several  gigabytes
       depending  on  the  compression  settings.  The settings used
when compressing a file
       determine the memory requirements of the decompressor.
Typically  the  decompressor
       needs  5 %  to  20 % of the amount of memory that the
compressor needed when creating
       the file.  For example, decompressing a file created with xz
-9  currently  requires
       65 MiB of memory.  Still, it is possible to have .xz files that
require several giga‐
       bytes of memory to decompress.


Three things come to mind:

1) You may not have enough memory free
2) There may be a bug (either compile/link-induced or code-induced) in
the copy of xz you're using
3) Upstream used some insane settings, causing a massive increase in
the amount of RAM required to decompress that stream.


You could download the .tar.xz file, decompress it on a different box,
and then recompress it with lighter settings.

unxz filename.tar.xz
xz -1 filename.tar

-- 
:wq

Re: [gentoo-user] xz memory hungry?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 491 bytes --]

On Wed, 22 Aug 2012 15:05:38 -0400, Michael Mol wrote:

> You could download the .tar.xz file, decompress it on a different box,
> and then recompress it with lighter settings.
> 
> unxz filename.tar.xz
> xz -1 filename.tar

You'd have to rebuild the ebuild's manifest after doing that, or portage
will download the original again.


-- 
Neil Bothwick

"Be strict when sending and tolerant when receiving."
 RFC 1958 - Architectural Principles of the Internet - section 3.9

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] xz memory hungry?

[Michael Mol]

On Wed, Aug 22, 2012 at 4:12 PM, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Wed, 22 Aug 2012 15:05:38 -0400, Michael Mol wrote:
>
>> You could download the .tar.xz file, decompress it on a different box,
>> and then recompress it with lighter settings.
>>
>> unxz filename.tar.xz
>> xz -1 filename.tar
>
> You'd have to rebuild the ebuild's manifest after doing that, or portage
> will download the original again.

Fair point.


-- 
:wq

Re: [gentoo-user] xz memory hungry?

[Jorge Almeida]

On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
>> xz: (stdin): Cannot allocate memory
>>
>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
>
> How much do you have free? From xz's manpage:

Almost all of it! It's a one-user workstation, which was essentialy idle.
>
>
I read the man page of xz, but it suggested nothing to me.
>
> Three things come to mind:
>
> 1) You may not have enough memory free
> 2) There may be a bug (either compile/link-induced or code-induced) in
> the copy of xz you're using
> 3) Upstream used some insane settings, causing a massive increase in
> the amount of RAM required to decompress that stream.
>
>
> You could download the .tar.xz file, decompress it on a different box,
> and then recompress it with lighter settings.
>
> unxz filename.tar.xz
> xz -1 filename.tar
>
Done that. It extracts now,  so 3) is the correct hypothesis, and  "insane" is
really the appropriate word. Of course, the hash digests are now wrong, so
emerge still fails. Any idea how to find which amount of memory is needed? I
would setup appropriate swap, if possible. The LFS site
http://www.linuxfromscratch.org/lfs/view/development/chapter03/packages.html
shows that there exists a tar.bz2 tarball. I think the ebuild should pull
that... Can't believe there are no gentooers out there with boxes with less ram.

Thanks a lot

Jorge Almeida

Re: [gentoo-user] xz memory hungry?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1599 bytes --]

Am 22.08.2012 22:32, schrieb Jorge Almeida:
> On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
>>> xz: (stdin): Cannot allocate memory
>>>
>>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
>>
>> How much do you have free? From xz's manpage:
> 
> Almost all of it! It's a one-user workstation, which was essentialy idle.
>>
>>
> I read the man page of xz, but it suggested nothing to me.
>>
>> Three things come to mind:
>>
>> 1) You may not have enough memory free
>> 2) There may be a bug (either compile/link-induced or code-induced) in
>> the copy of xz you're using
>> 3) Upstream used some insane settings, causing a massive increase in
>> the amount of RAM required to decompress that stream.
>>
>>
>> You could download the .tar.xz file, decompress it on a different box,
>> and then recompress it with lighter settings.
>>
>> unxz filename.tar.xz
>> xz -1 filename.tar
>>
> Done that. It extracts now,  so 3) is the correct hypothesis, and  "insane" is
> really the appropriate word. Of course, the hash digests are now wrong, so
> emerge still fails. Any idea how to find which amount of memory is needed? I
> would setup appropriate swap, if possible. [...]

There is a table in `man xz` showing the memory requirements. Even with
the highest setting, you only need 65 MB memory for decompression (674
MB for compression, though).

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] xz memory hungry?

[Michael Mol]

On Wed, Aug 22, 2012 at 4:43 PM, Florian Philipp <lists@binarywings.net> wrote:
> Am 22.08.2012 22:32, schrieb Jorge Almeida:
>> On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote:
>>> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
>>>> xz: (stdin): Cannot allocate memory
>>>>
>>>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
>>>
>>> How much do you have free? From xz's manpage:
>>
>> Almost all of it! It's a one-user workstation, which was essentialy idle.
>>>
>>>
>> I read the man page of xz, but it suggested nothing to me.
>>>
>>> Three things come to mind:
>>>
>>> 1) You may not have enough memory free
>>> 2) There may be a bug (either compile/link-induced or code-induced) in
>>> the copy of xz you're using
>>> 3) Upstream used some insane settings, causing a massive increase in
>>> the amount of RAM required to decompress that stream.
>>>
>>>
>>> You could download the .tar.xz file, decompress it on a different box,
>>> and then recompress it with lighter settings.
>>>
>>> unxz filename.tar.xz
>>> xz -1 filename.tar
>>>
>> Done that. It extracts now,  so 3) is the correct hypothesis, and  "insane" is
>> really the appropriate word. Of course, the hash digests are now wrong, so
>> emerge still fails. Any idea how to find which amount of memory is needed? I
>> would setup appropriate swap, if possible. [...]
>
> There is a table in `man xz` showing the memory requirements. Even with
> the highest setting, you only need 65 MB memory for decompression (674
> MB for compression, though).

Still not sure about this portion: "Still, it is possible to have .xz files that
require several giga‐bytes of memory to decompress."

But, yeah, this seems very wonky.

-- 
:wq

Re: [gentoo-user] xz memory hungry?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 377 bytes --]

On Wed, 22 Aug 2012 21:32:56 +0100, Jorge Almeida wrote:

> Done that. It extracts now,  so 3) is the correct hypothesis, and
> "insane" is really the appropriate word. Of course, the hash digests
> are now wrong, so emerge still fails.

sudo ebuild $PORTDIR/cat/pkg/pkg-ver.ebuild manifest


-- 
Neil Bothwick

The computer revolution is over. The computers won.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] xz memory hungry?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

Am 22.08.2012 20:52, schrieb Jorge Almeida:
> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
> xz: (stdin): Cannot allocate memory
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> 
> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
> Emerging fails on m4. tar xJvf fails both from within the chroot and from the
> host system. top shows that nothing is using any amount of memory worth
> mentioning. Extracting libtool-2.4.tar.xz works. I can extract
> m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
> related, except that I have no choice, as m4 is pulled by other packages.
> What to do?
> 
> app-arch/xz-utils-5.0.3 in chroot
> xz 5.0.4 in host system (Archlinux)
> 
> TIA
> 
> Jorge Almeida
> 

This should not happen, especially on such a small archive. I've tried
`strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
memory allocations). They never were larger than 68 MB

Try it yourself. The second parameter in mmap is the allocated size in byte.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] xz memory hungry?

[Jorge Almeida]

On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote:
> Am 22.08.2012 20:52, schrieb Jorge Almeida:
>
> This should not happen, especially on such a small archive. I've tried
> `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
> memory allocations). They never were larger than 68 MB
>
> Try it yourself. The second parameter in mmap is the allocated size in byte.
>
>
In the box where it works:

$ strace -e trace=mmap2 xz -t m4-1.4.16.tar.xz
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7746000
mmap2(NULL, 134226, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7725000
mmap2(NULL, 155888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0xb76fe000
mmap2(0xb7723000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24) = 0xb7723000
mmap2(NULL, 107004, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0xb76e3000
mmap2(0xb76fa000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0xb76fa000
mmap2(0xb76fc000, 4604, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76fc000
mmap2(NULL, 1727172, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xb753d000
mmap2(0xb76dd000, 12288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19f) = 0xb76dd000
mmap2(0xb76e0000, 10948, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76e0000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb753c000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb753b000
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb733b000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7745000
mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0xb723a000
+++ exited with 0 +++


In the other box, in the gentoo chroot:

# strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb779e000
mmap2(NULL, 12143, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb779b000
mmap2(NULL, 143600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0xb7777000
mmap2(0xb7799000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21) = 0xb7799000
mmap2(NULL, 1448488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xb7615000
mmap2(0xb7771000, 12288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb7771000
mmap2(0xb7774000, 10792, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7774000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7614000
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7414000
mmap2(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0x102a) = 0xb779d000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb779c000
mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 67244032, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2097152, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xb7214000
mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb779c000
xz: /usr/portage/distfiles/m4-1.4.16.tar.xz: Cannot allocate memory
+++ exited with 1 +++


Thanks,

Jorge Almeida

Re: [gentoo-user] xz memory hungry?

[Michael Mol]

On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote:
>> Am 22.08.2012 20:52, schrieb Jorge Almeida:
>>
>> This should not happen, especially on such a small archive. I've tried
>> `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
>> memory allocations). They never were larger than 68 MB
>>
>> Try it yourself. The second parameter in mmap is the allocated size in byte.
>>
>>
> In the box where it works:

[snip]
> In the other box, in the gentoo chroot:
>
> # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz

[snip]

> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = -1 ENOMEM (Cannot allocate memory)

It's only asking for about 65MB of RAM there. ENOMEM is nearly a
catch-all failure code for mmap().

My bet is that it's an incompatibility between the Arch kernel and the
version of glibc in the chroot.

-- 
:wq

Re: [gentoo-user] xz memory hungry?

[Michael Mol]

On Wed, Aug 22, 2012 at 5:42 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote:
>>> Am 22.08.2012 20:52, schrieb Jorge Almeida:
>>>
>>> This should not happen, especially on such a small archive. I've tried
>>> `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
>>> memory allocations). They never were larger than 68 MB
>>>
>>> Try it yourself. The second parameter in mmap is the allocated size in byte.
>>>
>>>
>> In the box where it works:
>
> [snip]
>> In the other box, in the gentoo chroot:
>>
>> # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz
>
> [snip]
>
>> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
>> -1, 0) = -1 ENOMEM (Cannot allocate memory)
>
> It's only asking for about 65MB of RAM there. ENOMEM is nearly a
> catch-all failure code for mmap().
>
> My bet is that it's an incompatibility between the Arch kernel and the
> version of glibc in the chroot.

(incidentally...hey! It's Gentoo-related after all. ;) )

-- 
:wq

Re: [gentoo-user] xz memory hungry?

[Jorge Almeida]

On Wed, Aug 22, 2012 at 10:42 PM, Michael Mol <mikemol@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote:
>>> Am 22.08.2012 20:52, schrieb Jorge Almeida:
>
> My bet is that it's an incompatibility between the Arch kernel and the
> version of glibc in the chroot.
>
Well, the kernel in Arch is 3.4.4 and glibc is 2.15. In the chroot, the glibc
is 2.14. Maybe I should update system et al. But then emerge system
asks for m4 (but not for glibc, which I find surprising).

Thanks,

J.

Re: [gentoo-user] xz memory hungry?

[Jorge Almeida]

On Wed, Aug 22, 2012 at 11:19 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 10:42 PM, Michael Mol <mikemol@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>
>> My bet is that it's an incompatibility between the Arch kernel and the
>> version of glibc in the chroot.
>>
> Well, the kernel in Arch is 3.4.4 and glibc is 2.15. In the chroot, the glibc
> is 2.14. Maybe I should update system et al. But then emerge system
> asks for m4 (but not for glibc, which I find surprising).
>
Well, I found the problem: ulimit problem. Not the first time this crap bites
me, but I always forget. I just wish this was better documented, somewhere.

Sorry for the noise, and thanks to everyone.

Jorge Almeida

Re: [gentoo-user] xz memory hungry?

[Michael Mol]

On Wed, Aug 22, 2012 at 8:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 11:19 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> On Wed, Aug 22, 2012 at 10:42 PM, Michael Mol <mikemol@gmail.com> wrote:
>>> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>>>
>>> My bet is that it's an incompatibility between the Arch kernel and the
>>> version of glibc in the chroot.
>>>
>> Well, the kernel in Arch is 3.4.4 and glibc is 2.15. In the chroot, the glibc
>> is 2.14. Maybe I should update system et al. But then emerge system
>> asks for m4 (but not for glibc, which I find surprising).
>>
> Well, I found the problem: ulimit problem. Not the first time this crap bites
> me, but I always forget. I just wish this was better documented, somewhere.
>
> Sorry for the noise, and thanks to everyone.

Hey, I learned something today. No complaints here. :)


-- 
:wq

Re: [gentoo-user] xz memory hungry?

[Paul Hartman]

On Wed, Aug 22, 2012 at 7:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> Well, I found the problem: ulimit problem. Not the first time this crap bites
> me, but I always forget. I just wish this was better documented, somewhere.

I tried to use ulimit to change stack size system-wide once, to reduce
RAM usage on 256M box, and it resulted in strange problems like this.
I changed it back to default and leave it alone since then except for
specific services because I don't fully understand the magic that
happens inside the box. :)

Re: [gentoo-user] xz memory hungry?

[Jorge Almeida]

On Thu, Aug 23, 2012 at 4:34 AM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Wed, Aug 22, 2012 at 7:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
>> Well, I found the problem: ulimit problem. Not the first time this crap bites
>> me, but I always forget. I just wish this was better documented, somewhere.
>
> I tried to use ulimit to change stack size system-wide once, to reduce
> RAM usage on 256M box, and it resulted in strange problems like this.
> I changed it back to default and leave it alone since then except for
> specific services because I don't fully understand the magic that
> happens inside the box. :)
>
Last time I had a problem like this I spent a lot of time googling about
ulimit/setting_limits/etc and found _nothing_ worth mentioning. This time I
run "ulimit -v unlimited", but the question is who put the former values
there?  Some hard-coded default? I couldn't find anything in init scripts nor
in bash rc files. I know that on logout the value is lost (I had to run ulimit
again on chrooting). What is the appropriate file to put "ulimit -v unlimited"
in? Perhaps ~/.bash_profile? And how can root set different hard limits for
different users? Maybe some bash guru will step in?:)

J.A.

Re: [gentoo-user] xz memory hungry?

[Bill Kenworthy]

On Thu, 2012-08-23 at 09:37 +0100, Jorge Almeida wrote:
> On Thu, Aug 23, 2012 at 4:34 AM, Paul Hartman
> <paul.hartman+gentoo@gmail.com> wrote:
> > On Wed, Aug 22, 2012 at 7:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote:
> >> Well, I found the problem: ulimit problem. Not the first time this crap bites
> >> me, but I always forget. I just wish this was better documented, somewhere.
> >
> > I tried to use ulimit to change stack size system-wide once, to reduce
> > RAM usage on 256M box, and it resulted in strange problems like this.
> > I changed it back to default and leave it alone since then except for
> > specific services because I don't fully understand the magic that
> > happens inside the box. :)
> >
> Last time I had a problem like this I spent a lot of time googling about
> ulimit/setting_limits/etc and found _nothing_ worth mentioning. This time I
> run "ulimit -v unlimited", but the question is who put the former values
> there?  Some hard-coded default? I couldn't find anything in init scripts nor
> in bash rc files. I know that on logout the value is lost (I had to run ulimit
> again on chrooting). What is the appropriate file to put "ulimit -v unlimited"
> in? Perhaps ~/.bash_profile? And how can root set different hard limits for
> different users? Maybe some bash guru will step in?:)
> 
> J.A.
> 

probably rc.conf, or maybe login.defs depending on per user/or everyone

BillK


troll ~ # grep limit /etc/*
/etc/freetds.conf:      # 'text size' to a more reasonable limit 
/etc/jwhois.conf:               rwhois-limit = 10;
/etc/login.defs:# Enable setting of ulimit, umask, and niceness from passwd gecos field.
/etc/login.defs:# a ":" delimited list of device names.  Root logins will be allowed only
/etc/login.defs:# If defined, ":" delimited list of "message of the day" files to
/etc/login.defs:#       ULIMIT          Default "ulimit" value.
/etc/login.defs:# (now it works with setrlimit too; ulimit is in 512-byte units)
/etc/login.defs:# It supports passwords of unlimited length and longer salt strings.
/etc/login.defs:# with the same group ID, to avoid limitation of the line length in the
/etc/lynx.cfg:# For instance, if SESSION_LIMIT is 250, a per-session limit of 250 entries of
/etc/lynx.cfg:# There is no fixed limit on the number of entries which can be restored;
/etc/lynx.cfg:# It is limited only by available memory.
/etc/lynx.cfg:# we need to limit the charset in outgoing mail to reduce
/etc/lynx.cfg:# The news reading facility in Lynx is quite limited.  Lynx does not provide a
/etc/lynx.cfg:# The posting facility in Lynx is quite limited.  Lynx does not provide a
/etc/lynx.cfg:# COOKIE_ACCEPT_DOMAINS and COOKIE_REJECT_DOMAINS are comma-delimited lists
/etc/lynx.cfg:# COOKIE_QUERY_INVALID_DOMAINS are comma-delimited lists of domains.
/etc/lynx.cfg:# MAX_COOKIES_BUFFER are limits on the total number of cookies for each domain,
/etc/lynx.cfg:# globally, and the per-cookie buffer size.  These limits are by default large
Binary file /etc/prelink.cache matches
/etc/rc.conf:# Pass ulimit parameters
/etc/rc.conf:#rc_ulimit="-u 30"
/etc/smartd.conf:#   -W D,I,C Monitor Temperature D)ifference, I)nformal limit, C)ritical limit
/etc/wgetrc:# default quota is unlimited.
troll ~ #

Re: [gentoo-user] xz memory hungry?

[Jorge Almeida]

On Thu, Aug 23, 2012 at 10:47 AM, Bill Kenworthy <billk@iinet.net.au> wrote:
>> >
>> Last time I had a problem like this I spent a lot of time googling about
>> ulimit/setting_limits/etc and found _nothing_ worth mentioning. This time I
>> run "ulimit -v unlimited", but the question is who put the former values
>> there?  Some hard-coded default? I couldn't find anything in init scripts nor
>> in bash rc files. I know that on logout the value is lost (I had to run ulimit
>> again on chrooting). What is the appropriate file to put "ulimit -v unlimited"
>> in? Perhaps ~/.bash_profile? And how can root set different hard limits for
>> different users? Maybe some bash guru will step in?:)
>>
>
> probably rc.conf, or maybe login.defs depending on per user/or everyone
>
It seems rc.conf has a variable for ulimit -u, not for the other flags.
login.defs has ULIMIT but teh man page just says "Default ulimit value"
Ah well...

Thanks

J.A.

Re: [gentoo-user] xz memory hungry?

[Blakawk]

On 23/08/2012 12:14, Jorge Almeida wrote:
> On Thu, Aug 23, 2012 at 10:47 AM, Bill Kenworthy <billk@iinet.net.au>
> wrote:
>
>>> Last time I had a problem like this I spent a lot of time googling
>>> about ulimit/setting_limits/etc and found _nothing_ worth 
>>> mentioning.
>>> This time I run "ulimit -v unlimited", but the question is who put
>>> the former values there? Some hard-coded default? I couldn't find
>>> anything in init scripts nor in bash rc files. I know that on 
>>> logout
>>> the value is lost (I had to run ulimit again on chrooting). What is
>>> the appropriate file to put "ulimit -v unlimited" in? Perhaps
>>> ~/.bash_profile? And how can root set different hard limits for
>>> different users? Maybe some bash guru will step in?:)
>> probably rc.conf, or maybe login.defs depending on per user/or
>> everyone
>
> It seems rc.conf has a variable for ulimit -u, not for the other 
> flags.
> login.defs has ULIMIT but teh man page just says "Default ulimit 
> value"
> Ah well...

In /etc/security/limits.conf you can put any limits that can be set 
using ulimit command so they are kept between reboots.

> Thanks
>
> J.A.

Re: [gentoo-user] xz memory hungry?

[Jorge Almeida]

On Thu, Aug 23, 2012 at 12:23 PM, Blakawk <blakawk@gentooist.com> wrote:
> On 23/08/2012 12:14, Jorge Almeida wrote:

>
>
> In /etc/security/limits.conf you can put any limits that can be set using
> ulimit command so they are kept between reboots.
>
OK, but I'm not using PAM for anything, so the question remains of how the
values are currently set.


 Thanks

 J.A.

Re: [gentoo-user] xz memory hungry?

[Volker Armin Hemmann]

Am Mittwoch, 22. August 2012, 22:16:19 schrieb Jorge Almeida:
> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> 
wrote:
> > Am 22.08.2012 20:52, schrieb Jorge Almeida:
> > 
> > This should not happen, especially on such a small archive. I've tried
> > `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g.
> > memory allocations). They never were larger than 68 MB
> > 
> > Try it yourself. The second parameter in mmap is the allocated size in
> > byte.
> In the box where it works:
> 
> $ strace -e trace=mmap2 xz -t m4-1.4.16.tar.xz
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb7746000
> mmap2(NULL, 134226, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7725000
> mmap2(NULL, 155888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0xb76fe000
> mmap2(0xb7723000, 8192, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24) = 0xb7723000
> mmap2(NULL, 107004, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0xb76e3000
> mmap2(0xb76fa000, 8192, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0xb76fa000
> mmap2(0xb76fc000, 4604, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76fc000
> mmap2(NULL, 1727172, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
> 3, 0) = 0xb753d000
> mmap2(0xb76dd000, 12288, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19f) = 0xb76dd000
> mmap2(0xb76e0000, 10948, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76e0000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb753c000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb753b000
> mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb733b000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb7745000
> mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = 0xb723a000
> +++ exited with 0 +++
> 
> 
> In the other box, in the gentoo chroot:
> 
> # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb779e000
> mmap2(NULL, 12143, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb779b000
> mmap2(NULL, 143600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
> 0) = 0xb7777000
> mmap2(0xb7799000, 8192, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21) = 0xb7799000
> mmap2(NULL, 1448488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
> 3, 0) = 0xb7615000
> mmap2(0xb7771000, 12288, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb7771000
> mmap2(0xb7774000, 10792, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7774000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb7614000
> mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7414000
> mmap2(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0x102a) = 0xb779d000
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb779c000
> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = -1 ENOMEM (Cannot allocate memory)
> mmap2(NULL, 67244032, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = -1 ENOMEM (Cannot allocate memory)
> mmap2(NULL, 2097152, PROT_NONE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xb7214000
> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
> -1, 0) = -1 ENOMEM (Cannot allocate memory)
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb779c000
> xz: /usr/portage/distfiles/m4-1.4.16.tar.xz: Cannot allocate memory
> +++ exited with 1 +++
> 
> 
> Thanks,
> 
> Jorge Almeida


maybe a memory fragmentation problem? It tries to allocate 65mb continous 
chunk - and can't find one?
-- 
#163933

[gentoo-user] Re: xz memory hungry?

[Nikos Chantziaras]

On 22/08/12 21:52, Jorge Almeida wrote:
> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz
> xz: (stdin): Cannot allocate memory
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro.
> Emerging fails on m4. tar xJvf fails both from within the chroot and from the
> host system. top shows that nothing is using any amount of memory worth
> mentioning. Extracting libtool-2.4.tar.xz works. I can extract
> m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo
> related, except that I have no choice, as m4 is pulled by other packages.
> What to do?
>
> app-arch/xz-utils-5.0.3 in chroot
> xz 5.0.4 in host system (Archlinux)

You should file a bug about this.  Whoever put that xz there surely has 
no idea that this is happening.

Re: [gentoo-user] Re: xz memory hungry?

[Jorge Almeida]

On Thu, Aug 23, 2012 at 2:43 AM, Nikos Chantziaras <realnc@gmail.com> wrote:
> On 22/08/12 21:52, Jorge Almeida wrote:
>>
>
>
> You should file a bug about this.  Whoever put that xz there surely has no
> idea that this is happening.
>
>
Not Gentoo nor xz fault (see other messages in thread).

Thanks

J.A.