* [gentoo-user] xz memory hungry? @ 2012-08-22 18:52 Jorge Almeida 2012-08-22 19:05 ` Michael Mol ` (2 more replies) 0 siblings, 3 replies; 24+ messages in thread From: Jorge Almeida @ 2012-08-22 18:52 UTC (permalink / raw To: gentoo-user # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz xz: (stdin): Cannot allocate memory tar: Child returned status 1 tar: Error is not recoverable: exiting now The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro. Emerging fails on m4. tar xJvf fails both from within the chroot and from the host system. top shows that nothing is using any amount of memory worth mentioning. Extracting libtool-2.4.tar.xz works. I can extract m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo related, except that I have no choice, as m4 is pulled by other packages. What to do? app-arch/xz-utils-5.0.3 in chroot xz 5.0.4 in host system (Archlinux) TIA Jorge Almeida ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 18:52 [gentoo-user] xz memory hungry? Jorge Almeida @ 2012-08-22 19:05 ` Michael Mol 2012-08-22 20:12 ` Neil Bothwick 2012-08-22 20:32 ` Jorge Almeida 2012-08-22 20:39 ` Florian Philipp 2012-08-23 1:43 ` [gentoo-user] " Nikos Chantziaras 2 siblings, 2 replies; 24+ messages in thread From: Michael Mol @ 2012-08-22 19:05 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: > # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz > xz: (stdin): Cannot allocate memory > tar: Child returned status 1 > tar: Error is not recoverable: exiting now > > The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro. > Emerging fails on m4. tar xJvf fails both from within the chroot and from the > host system. top shows that nothing is using any amount of memory worth > mentioning. Extracting libtool-2.4.tar.xz works. I can extract > m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo > related, except that I have no choice, as m4 is pulled by other packages. > What to do? > > app-arch/xz-utils-5.0.3 in chroot > xz 5.0.4 in host system (Archlinux) How much do you have free? From xz's manpage: Memory usage The memory usage of xz varies from a few hundred kilobytes to several gigabytes depending on the compression settings. The settings used when compressing a file determine the memory requirements of the decompressor. Typically the decompressor needs 5 % to 20 % of the amount of memory that the compressor needed when creating the file. For example, decompressing a file created with xz -9 currently requires 65 MiB of memory. Still, it is possible to have .xz files that require several giga‐ bytes of memory to decompress. Three things come to mind: 1) You may not have enough memory free 2) There may be a bug (either compile/link-induced or code-induced) in the copy of xz you're using 3) Upstream used some insane settings, causing a massive increase in the amount of RAM required to decompress that stream. You could download the .tar.xz file, decompress it on a different box, and then recompress it with lighter settings. unxz filename.tar.xz xz -1 filename.tar -- :wq ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 19:05 ` Michael Mol @ 2012-08-22 20:12 ` Neil Bothwick 2012-08-22 20:24 ` Michael Mol 2012-08-22 20:32 ` Jorge Almeida 1 sibling, 1 reply; 24+ messages in thread From: Neil Bothwick @ 2012-08-22 20:12 UTC (permalink / raw To: gentoo-user [-- Attachment #1: Type: text/plain, Size: 491 bytes --] On Wed, 22 Aug 2012 15:05:38 -0400, Michael Mol wrote: > You could download the .tar.xz file, decompress it on a different box, > and then recompress it with lighter settings. > > unxz filename.tar.xz > xz -1 filename.tar You'd have to rebuild the ebuild's manifest after doing that, or portage will download the original again. -- Neil Bothwick "Be strict when sending and tolerant when receiving." RFC 1958 - Architectural Principles of the Internet - section 3.9 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 20:12 ` Neil Bothwick @ 2012-08-22 20:24 ` Michael Mol 0 siblings, 0 replies; 24+ messages in thread From: Michael Mol @ 2012-08-22 20:24 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 4:12 PM, Neil Bothwick <neil@digimed.co.uk> wrote: > On Wed, 22 Aug 2012 15:05:38 -0400, Michael Mol wrote: > >> You could download the .tar.xz file, decompress it on a different box, >> and then recompress it with lighter settings. >> >> unxz filename.tar.xz >> xz -1 filename.tar > > You'd have to rebuild the ebuild's manifest after doing that, or portage > will download the original again. Fair point. -- :wq ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 19:05 ` Michael Mol 2012-08-22 20:12 ` Neil Bothwick @ 2012-08-22 20:32 ` Jorge Almeida 2012-08-22 20:43 ` Florian Philipp 2012-08-22 21:10 ` Neil Bothwick 1 sibling, 2 replies; 24+ messages in thread From: Jorge Almeida @ 2012-08-22 20:32 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote: > On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: >> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz >> xz: (stdin): Cannot allocate memory >> >> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro. > > How much do you have free? From xz's manpage: Almost all of it! It's a one-user workstation, which was essentialy idle. > > I read the man page of xz, but it suggested nothing to me. > > Three things come to mind: > > 1) You may not have enough memory free > 2) There may be a bug (either compile/link-induced or code-induced) in > the copy of xz you're using > 3) Upstream used some insane settings, causing a massive increase in > the amount of RAM required to decompress that stream. > > > You could download the .tar.xz file, decompress it on a different box, > and then recompress it with lighter settings. > > unxz filename.tar.xz > xz -1 filename.tar > Done that. It extracts now, so 3) is the correct hypothesis, and "insane" is really the appropriate word. Of course, the hash digests are now wrong, so emerge still fails. Any idea how to find which amount of memory is needed? I would setup appropriate swap, if possible. The LFS site http://www.linuxfromscratch.org/lfs/view/development/chapter03/packages.html shows that there exists a tar.bz2 tarball. I think the ebuild should pull that... Can't believe there are no gentooers out there with boxes with less ram. Thanks a lot Jorge Almeida ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 20:32 ` Jorge Almeida @ 2012-08-22 20:43 ` Florian Philipp 2012-08-22 21:12 ` Michael Mol 2012-08-22 21:10 ` Neil Bothwick 1 sibling, 1 reply; 24+ messages in thread From: Florian Philipp @ 2012-08-22 20:43 UTC (permalink / raw To: gentoo-user [-- Attachment #1: Type: text/plain, Size: 1599 bytes --] Am 22.08.2012 22:32, schrieb Jorge Almeida: > On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote: >> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: >>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz >>> xz: (stdin): Cannot allocate memory >>> >>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro. >> >> How much do you have free? From xz's manpage: > > Almost all of it! It's a one-user workstation, which was essentialy idle. >> >> > I read the man page of xz, but it suggested nothing to me. >> >> Three things come to mind: >> >> 1) You may not have enough memory free >> 2) There may be a bug (either compile/link-induced or code-induced) in >> the copy of xz you're using >> 3) Upstream used some insane settings, causing a massive increase in >> the amount of RAM required to decompress that stream. >> >> >> You could download the .tar.xz file, decompress it on a different box, >> and then recompress it with lighter settings. >> >> unxz filename.tar.xz >> xz -1 filename.tar >> > Done that. It extracts now, so 3) is the correct hypothesis, and "insane" is > really the appropriate word. Of course, the hash digests are now wrong, so > emerge still fails. Any idea how to find which amount of memory is needed? I > would setup appropriate swap, if possible. [...] There is a table in `man xz` showing the memory requirements. Even with the highest setting, you only need 65 MB memory for decompression (674 MB for compression, though). Regards, Florian Philipp [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 262 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 20:43 ` Florian Philipp @ 2012-08-22 21:12 ` Michael Mol 0 siblings, 0 replies; 24+ messages in thread From: Michael Mol @ 2012-08-22 21:12 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 4:43 PM, Florian Philipp <lists@binarywings.net> wrote: > Am 22.08.2012 22:32, schrieb Jorge Almeida: >> On Wed, Aug 22, 2012 at 8:05 PM, Michael Mol <mikemol@gmail.com> wrote: >>> On Wed, Aug 22, 2012 at 2:52 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: >>>> # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz >>>> xz: (stdin): Cannot allocate memory >>>> >>>> The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro. >>> >>> How much do you have free? From xz's manpage: >> >> Almost all of it! It's a one-user workstation, which was essentialy idle. >>> >>> >> I read the man page of xz, but it suggested nothing to me. >>> >>> Three things come to mind: >>> >>> 1) You may not have enough memory free >>> 2) There may be a bug (either compile/link-induced or code-induced) in >>> the copy of xz you're using >>> 3) Upstream used some insane settings, causing a massive increase in >>> the amount of RAM required to decompress that stream. >>> >>> >>> You could download the .tar.xz file, decompress it on a different box, >>> and then recompress it with lighter settings. >>> >>> unxz filename.tar.xz >>> xz -1 filename.tar >>> >> Done that. It extracts now, so 3) is the correct hypothesis, and "insane" is >> really the appropriate word. Of course, the hash digests are now wrong, so >> emerge still fails. Any idea how to find which amount of memory is needed? I >> would setup appropriate swap, if possible. [...] > > There is a table in `man xz` showing the memory requirements. Even with > the highest setting, you only need 65 MB memory for decompression (674 > MB for compression, though). Still not sure about this portion: "Still, it is possible to have .xz files that require several giga‐bytes of memory to decompress." But, yeah, this seems very wonky. -- :wq ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 20:32 ` Jorge Almeida 2012-08-22 20:43 ` Florian Philipp @ 2012-08-22 21:10 ` Neil Bothwick 1 sibling, 0 replies; 24+ messages in thread From: Neil Bothwick @ 2012-08-22 21:10 UTC (permalink / raw To: gentoo-user [-- Attachment #1: Type: text/plain, Size: 377 bytes --] On Wed, 22 Aug 2012 21:32:56 +0100, Jorge Almeida wrote: > Done that. It extracts now, so 3) is the correct hypothesis, and > "insane" is really the appropriate word. Of course, the hash digests > are now wrong, so emerge still fails. sudo ebuild $PORTDIR/cat/pkg/pkg-ver.ebuild manifest -- Neil Bothwick The computer revolution is over. The computers won. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 18:52 [gentoo-user] xz memory hungry? Jorge Almeida 2012-08-22 19:05 ` Michael Mol @ 2012-08-22 20:39 ` Florian Philipp 2012-08-22 21:16 ` Jorge Almeida 2012-08-23 1:43 ` [gentoo-user] " Nikos Chantziaras 2 siblings, 1 reply; 24+ messages in thread From: Florian Philipp @ 2012-08-22 20:39 UTC (permalink / raw To: gentoo-user [-- Attachment #1: Type: text/plain, Size: 1118 bytes --] Am 22.08.2012 20:52, schrieb Jorge Almeida: > # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz > xz: (stdin): Cannot allocate memory > tar: Child returned status 1 > tar: Error is not recoverable: exiting now > > The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro. > Emerging fails on m4. tar xJvf fails both from within the chroot and from the > host system. top shows that nothing is using any amount of memory worth > mentioning. Extracting libtool-2.4.tar.xz works. I can extract > m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo > related, except that I have no choice, as m4 is pulled by other packages. > What to do? > > app-arch/xz-utils-5.0.3 in chroot > xz 5.0.4 in host system (Archlinux) > > TIA > > Jorge Almeida > This should not happen, especially on such a small archive. I've tried `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g. memory allocations). They never were larger than 68 MB Try it yourself. The second parameter in mmap is the allocated size in byte. Regards, Florian Philipp [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 262 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 20:39 ` Florian Philipp @ 2012-08-22 21:16 ` Jorge Almeida 2012-08-22 21:42 ` Michael Mol 2012-08-23 8:22 ` Volker Armin Hemmann 0 siblings, 2 replies; 24+ messages in thread From: Jorge Almeida @ 2012-08-22 21:16 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote: > Am 22.08.2012 20:52, schrieb Jorge Almeida: > > This should not happen, especially on such a small archive. I've tried > `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g. > memory allocations). They never were larger than 68 MB > > Try it yourself. The second parameter in mmap is the allocated size in byte. > > In the box where it works: $ strace -e trace=mmap2 xz -t m4-1.4.16.tar.xz mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7746000 mmap2(NULL, 134226, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7725000 mmap2(NULL, 155888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb76fe000 mmap2(0xb7723000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24) = 0xb7723000 mmap2(NULL, 107004, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb76e3000 mmap2(0xb76fa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0xb76fa000 mmap2(0xb76fc000, 4604, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76fc000 mmap2(NULL, 1727172, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb753d000 mmap2(0xb76dd000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19f) = 0xb76dd000 mmap2(0xb76e0000, 10948, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76e0000 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb753c000 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb753b000 mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb733b000 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7745000 mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb723a000 +++ exited with 0 +++ In the other box, in the gentoo chroot: # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb779e000 mmap2(NULL, 12143, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb779b000 mmap2(NULL, 143600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7777000 mmap2(0xb7799000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21) = 0xb7799000 mmap2(NULL, 1448488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7615000 mmap2(0xb7771000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb7771000 mmap2(0xb7774000, 10792, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7774000 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7614000 mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7414000 mmap2(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0x102a) = 0xb779d000 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb779c000 mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) mmap2(NULL, 67244032, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xb7214000 mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory) mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb779c000 xz: /usr/portage/distfiles/m4-1.4.16.tar.xz: Cannot allocate memory +++ exited with 1 +++ Thanks, Jorge Almeida ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 21:16 ` Jorge Almeida @ 2012-08-22 21:42 ` Michael Mol 2012-08-22 21:42 ` Michael Mol 2012-08-22 22:19 ` Jorge Almeida 2012-08-23 8:22 ` Volker Armin Hemmann 1 sibling, 2 replies; 24+ messages in thread From: Michael Mol @ 2012-08-22 21:42 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: > On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote: >> Am 22.08.2012 20:52, schrieb Jorge Almeida: >> >> This should not happen, especially on such a small archive. I've tried >> `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g. >> memory allocations). They never were larger than 68 MB >> >> Try it yourself. The second parameter in mmap is the allocated size in byte. >> >> > In the box where it works: [snip] > In the other box, in the gentoo chroot: > > # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz [snip] > mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, > -1, 0) = -1 ENOMEM (Cannot allocate memory) It's only asking for about 65MB of RAM there. ENOMEM is nearly a catch-all failure code for mmap(). My bet is that it's an incompatibility between the Arch kernel and the version of glibc in the chroot. -- :wq ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 21:42 ` Michael Mol @ 2012-08-22 21:42 ` Michael Mol 2012-08-22 22:19 ` Jorge Almeida 1 sibling, 0 replies; 24+ messages in thread From: Michael Mol @ 2012-08-22 21:42 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 5:42 PM, Michael Mol <mikemol@gmail.com> wrote: > On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: >> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote: >>> Am 22.08.2012 20:52, schrieb Jorge Almeida: >>> >>> This should not happen, especially on such a small archive. I've tried >>> `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g. >>> memory allocations). They never were larger than 68 MB >>> >>> Try it yourself. The second parameter in mmap is the allocated size in byte. >>> >>> >> In the box where it works: > > [snip] >> In the other box, in the gentoo chroot: >> >> # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz > > [snip] > >> mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, >> -1, 0) = -1 ENOMEM (Cannot allocate memory) > > It's only asking for about 65MB of RAM there. ENOMEM is nearly a > catch-all failure code for mmap(). > > My bet is that it's an incompatibility between the Arch kernel and the > version of glibc in the chroot. (incidentally...hey! It's Gentoo-related after all. ;) ) -- :wq ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 21:42 ` Michael Mol 2012-08-22 21:42 ` Michael Mol @ 2012-08-22 22:19 ` Jorge Almeida 2012-08-23 0:24 ` Jorge Almeida 1 sibling, 1 reply; 24+ messages in thread From: Jorge Almeida @ 2012-08-22 22:19 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 10:42 PM, Michael Mol <mikemol@gmail.com> wrote: > On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: >> On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote: >>> Am 22.08.2012 20:52, schrieb Jorge Almeida: > > My bet is that it's an incompatibility between the Arch kernel and the > version of glibc in the chroot. > Well, the kernel in Arch is 3.4.4 and glibc is 2.15. In the chroot, the glibc is 2.14. Maybe I should update system et al. But then emerge system asks for m4 (but not for glibc, which I find surprising). Thanks, J. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 22:19 ` Jorge Almeida @ 2012-08-23 0:24 ` Jorge Almeida 2012-08-23 0:56 ` Michael Mol 2012-08-23 3:34 ` Paul Hartman 0 siblings, 2 replies; 24+ messages in thread From: Jorge Almeida @ 2012-08-23 0:24 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 11:19 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: > On Wed, Aug 22, 2012 at 10:42 PM, Michael Mol <mikemol@gmail.com> wrote: >> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: >> >> My bet is that it's an incompatibility between the Arch kernel and the >> version of glibc in the chroot. >> > Well, the kernel in Arch is 3.4.4 and glibc is 2.15. In the chroot, the glibc > is 2.14. Maybe I should update system et al. But then emerge system > asks for m4 (but not for glibc, which I find surprising). > Well, I found the problem: ulimit problem. Not the first time this crap bites me, but I always forget. I just wish this was better documented, somewhere. Sorry for the noise, and thanks to everyone. Jorge Almeida ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-23 0:24 ` Jorge Almeida @ 2012-08-23 0:56 ` Michael Mol 2012-08-23 3:34 ` Paul Hartman 1 sibling, 0 replies; 24+ messages in thread From: Michael Mol @ 2012-08-23 0:56 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 8:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: > On Wed, Aug 22, 2012 at 11:19 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: >> On Wed, Aug 22, 2012 at 10:42 PM, Michael Mol <mikemol@gmail.com> wrote: >>> On Wed, Aug 22, 2012 at 5:16 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: >>> >>> My bet is that it's an incompatibility between the Arch kernel and the >>> version of glibc in the chroot. >>> >> Well, the kernel in Arch is 3.4.4 and glibc is 2.15. In the chroot, the glibc >> is 2.14. Maybe I should update system et al. But then emerge system >> asks for m4 (but not for glibc, which I find surprising). >> > Well, I found the problem: ulimit problem. Not the first time this crap bites > me, but I always forget. I just wish this was better documented, somewhere. > > Sorry for the noise, and thanks to everyone. Hey, I learned something today. No complaints here. :) -- :wq ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-23 0:24 ` Jorge Almeida 2012-08-23 0:56 ` Michael Mol @ 2012-08-23 3:34 ` Paul Hartman 2012-08-23 8:37 ` Jorge Almeida 1 sibling, 1 reply; 24+ messages in thread From: Paul Hartman @ 2012-08-23 3:34 UTC (permalink / raw To: gentoo-user On Wed, Aug 22, 2012 at 7:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: > Well, I found the problem: ulimit problem. Not the first time this crap bites > me, but I always forget. I just wish this was better documented, somewhere. I tried to use ulimit to change stack size system-wide once, to reduce RAM usage on 256M box, and it resulted in strange problems like this. I changed it back to default and leave it alone since then except for specific services because I don't fully understand the magic that happens inside the box. :) ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-23 3:34 ` Paul Hartman @ 2012-08-23 8:37 ` Jorge Almeida 2012-08-23 9:47 ` Bill Kenworthy 0 siblings, 1 reply; 24+ messages in thread From: Jorge Almeida @ 2012-08-23 8:37 UTC (permalink / raw To: gentoo-user On Thu, Aug 23, 2012 at 4:34 AM, Paul Hartman <paul.hartman+gentoo@gmail.com> wrote: > On Wed, Aug 22, 2012 at 7:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: >> Well, I found the problem: ulimit problem. Not the first time this crap bites >> me, but I always forget. I just wish this was better documented, somewhere. > > I tried to use ulimit to change stack size system-wide once, to reduce > RAM usage on 256M box, and it resulted in strange problems like this. > I changed it back to default and leave it alone since then except for > specific services because I don't fully understand the magic that > happens inside the box. :) > Last time I had a problem like this I spent a lot of time googling about ulimit/setting_limits/etc and found _nothing_ worth mentioning. This time I run "ulimit -v unlimited", but the question is who put the former values there? Some hard-coded default? I couldn't find anything in init scripts nor in bash rc files. I know that on logout the value is lost (I had to run ulimit again on chrooting). What is the appropriate file to put "ulimit -v unlimited" in? Perhaps ~/.bash_profile? And how can root set different hard limits for different users? Maybe some bash guru will step in?:) J.A. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-23 8:37 ` Jorge Almeida @ 2012-08-23 9:47 ` Bill Kenworthy 2012-08-23 10:14 ` Jorge Almeida 0 siblings, 1 reply; 24+ messages in thread From: Bill Kenworthy @ 2012-08-23 9:47 UTC (permalink / raw To: gentoo-user On Thu, 2012-08-23 at 09:37 +0100, Jorge Almeida wrote: > On Thu, Aug 23, 2012 at 4:34 AM, Paul Hartman > <paul.hartman+gentoo@gmail.com> wrote: > > On Wed, Aug 22, 2012 at 7:24 PM, Jorge Almeida <jjalmeida@gmail.com> wrote: > >> Well, I found the problem: ulimit problem. Not the first time this crap bites > >> me, but I always forget. I just wish this was better documented, somewhere. > > > > I tried to use ulimit to change stack size system-wide once, to reduce > > RAM usage on 256M box, and it resulted in strange problems like this. > > I changed it back to default and leave it alone since then except for > > specific services because I don't fully understand the magic that > > happens inside the box. :) > > > Last time I had a problem like this I spent a lot of time googling about > ulimit/setting_limits/etc and found _nothing_ worth mentioning. This time I > run "ulimit -v unlimited", but the question is who put the former values > there? Some hard-coded default? I couldn't find anything in init scripts nor > in bash rc files. I know that on logout the value is lost (I had to run ulimit > again on chrooting). What is the appropriate file to put "ulimit -v unlimited" > in? Perhaps ~/.bash_profile? And how can root set different hard limits for > different users? Maybe some bash guru will step in?:) > > J.A. > probably rc.conf, or maybe login.defs depending on per user/or everyone BillK troll ~ # grep limit /etc/* /etc/freetds.conf: # 'text size' to a more reasonable limit /etc/jwhois.conf: rwhois-limit = 10; /etc/login.defs:# Enable setting of ulimit, umask, and niceness from passwd gecos field. /etc/login.defs:# a ":" delimited list of device names. Root logins will be allowed only /etc/login.defs:# If defined, ":" delimited list of "message of the day" files to /etc/login.defs:# ULIMIT Default "ulimit" value. /etc/login.defs:# (now it works with setrlimit too; ulimit is in 512-byte units) /etc/login.defs:# It supports passwords of unlimited length and longer salt strings. /etc/login.defs:# with the same group ID, to avoid limitation of the line length in the /etc/lynx.cfg:# For instance, if SESSION_LIMIT is 250, a per-session limit of 250 entries of /etc/lynx.cfg:# There is no fixed limit on the number of entries which can be restored; /etc/lynx.cfg:# It is limited only by available memory. /etc/lynx.cfg:# we need to limit the charset in outgoing mail to reduce /etc/lynx.cfg:# The news reading facility in Lynx is quite limited. Lynx does not provide a /etc/lynx.cfg:# The posting facility in Lynx is quite limited. Lynx does not provide a /etc/lynx.cfg:# COOKIE_ACCEPT_DOMAINS and COOKIE_REJECT_DOMAINS are comma-delimited lists /etc/lynx.cfg:# COOKIE_QUERY_INVALID_DOMAINS are comma-delimited lists of domains. /etc/lynx.cfg:# MAX_COOKIES_BUFFER are limits on the total number of cookies for each domain, /etc/lynx.cfg:# globally, and the per-cookie buffer size. These limits are by default large Binary file /etc/prelink.cache matches /etc/rc.conf:# Pass ulimit parameters /etc/rc.conf:#rc_ulimit="-u 30" /etc/smartd.conf:# -W D,I,C Monitor Temperature D)ifference, I)nformal limit, C)ritical limit /etc/wgetrc:# default quota is unlimited. troll ~ # ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-23 9:47 ` Bill Kenworthy @ 2012-08-23 10:14 ` Jorge Almeida 2012-08-23 11:23 ` Blakawk 0 siblings, 1 reply; 24+ messages in thread From: Jorge Almeida @ 2012-08-23 10:14 UTC (permalink / raw To: gentoo-user On Thu, Aug 23, 2012 at 10:47 AM, Bill Kenworthy <billk@iinet.net.au> wrote: >> > >> Last time I had a problem like this I spent a lot of time googling about >> ulimit/setting_limits/etc and found _nothing_ worth mentioning. This time I >> run "ulimit -v unlimited", but the question is who put the former values >> there? Some hard-coded default? I couldn't find anything in init scripts nor >> in bash rc files. I know that on logout the value is lost (I had to run ulimit >> again on chrooting). What is the appropriate file to put "ulimit -v unlimited" >> in? Perhaps ~/.bash_profile? And how can root set different hard limits for >> different users? Maybe some bash guru will step in?:) >> > > probably rc.conf, or maybe login.defs depending on per user/or everyone > It seems rc.conf has a variable for ulimit -u, not for the other flags. login.defs has ULIMIT but teh man page just says "Default ulimit value" Ah well... Thanks J.A. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-23 10:14 ` Jorge Almeida @ 2012-08-23 11:23 ` Blakawk 2012-08-23 11:37 ` Jorge Almeida 0 siblings, 1 reply; 24+ messages in thread From: Blakawk @ 2012-08-23 11:23 UTC (permalink / raw To: gentoo-user On 23/08/2012 12:14, Jorge Almeida wrote: > On Thu, Aug 23, 2012 at 10:47 AM, Bill Kenworthy <billk@iinet.net.au> > wrote: > >>> Last time I had a problem like this I spent a lot of time googling >>> about ulimit/setting_limits/etc and found _nothing_ worth >>> mentioning. >>> This time I run "ulimit -v unlimited", but the question is who put >>> the former values there? Some hard-coded default? I couldn't find >>> anything in init scripts nor in bash rc files. I know that on >>> logout >>> the value is lost (I had to run ulimit again on chrooting). What is >>> the appropriate file to put "ulimit -v unlimited" in? Perhaps >>> ~/.bash_profile? And how can root set different hard limits for >>> different users? Maybe some bash guru will step in?:) >> probably rc.conf, or maybe login.defs depending on per user/or >> everyone > > It seems rc.conf has a variable for ulimit -u, not for the other > flags. > login.defs has ULIMIT but teh man page just says "Default ulimit > value" > Ah well... In /etc/security/limits.conf you can put any limits that can be set using ulimit command so they are kept between reboots. > Thanks > > J.A. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-23 11:23 ` Blakawk @ 2012-08-23 11:37 ` Jorge Almeida 0 siblings, 0 replies; 24+ messages in thread From: Jorge Almeida @ 2012-08-23 11:37 UTC (permalink / raw To: gentoo-user On Thu, Aug 23, 2012 at 12:23 PM, Blakawk <blakawk@gentooist.com> wrote: > On 23/08/2012 12:14, Jorge Almeida wrote: > > > In /etc/security/limits.conf you can put any limits that can be set using > ulimit command so they are kept between reboots. > OK, but I'm not using PAM for anything, so the question remains of how the values are currently set. Thanks J.A. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] xz memory hungry? 2012-08-22 21:16 ` Jorge Almeida 2012-08-22 21:42 ` Michael Mol @ 2012-08-23 8:22 ` Volker Armin Hemmann 1 sibling, 0 replies; 24+ messages in thread From: Volker Armin Hemmann @ 2012-08-23 8:22 UTC (permalink / raw To: gentoo-user; +Cc: Jorge Almeida Am Mittwoch, 22. August 2012, 22:16:19 schrieb Jorge Almeida: > On Wed, Aug 22, 2012 at 9:39 PM, Florian Philipp <lists@binarywings.net> wrote: > > Am 22.08.2012 20:52, schrieb Jorge Almeida: > > > > This should not happen, especially on such a small archive. I've tried > > `strace xz -t m4-1.4.16.tar.xz` and looked for calls to mmap (e.g. > > memory allocations). They never were larger than 68 MB > > > > Try it yourself. The second parameter in mmap is the allocated size in > > byte. > In the box where it works: > > $ strace -e trace=mmap2 xz -t m4-1.4.16.tar.xz > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0xb7746000 > mmap2(NULL, 134226, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7725000 > mmap2(NULL, 155888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, > 0) = 0xb76fe000 > mmap2(0xb7723000, 8192, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24) = 0xb7723000 > mmap2(NULL, 107004, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, > 0) = 0xb76e3000 > mmap2(0xb76fa000, 8192, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0xb76fa000 > mmap2(0xb76fc000, 4604, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76fc000 > mmap2(NULL, 1727172, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, > 3, 0) = 0xb753d000 > mmap2(0xb76dd000, 12288, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19f) = 0xb76dd000 > mmap2(0xb76e0000, 10948, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb76e0000 > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0xb753c000 > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0xb753b000 > mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb733b000 > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0xb7745000 > mmap2(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, > -1, 0) = 0xb723a000 > +++ exited with 0 +++ > > > In the other box, in the gentoo chroot: > > # strace -e trace=mmap2 xz -t /usr/portage/distfiles/m4-1.4.16.tar.xz > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0xb779e000 > mmap2(NULL, 12143, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb779b000 > mmap2(NULL, 143600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, > 0) = 0xb7777000 > mmap2(0xb7799000, 8192, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21) = 0xb7799000 > mmap2(NULL, 1448488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, > 3, 0) = 0xb7615000 > mmap2(0xb7771000, 12288, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15c) = 0xb7771000 > mmap2(0xb7774000, 10792, PROT_READ|PROT_WRITE, > MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7774000 > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0xb7614000 > mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7414000 > mmap2(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0x102a) = 0xb779d000 > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0xb779c000 > mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, > -1, 0) = -1 ENOMEM (Cannot allocate memory) > mmap2(NULL, 67244032, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, > -1, 0) = -1 ENOMEM (Cannot allocate memory) > mmap2(NULL, 2097152, PROT_NONE, > MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xb7214000 > mmap2(NULL, 67112960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, > -1, 0) = -1 ENOMEM (Cannot allocate memory) > mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0xb779c000 > xz: /usr/portage/distfiles/m4-1.4.16.tar.xz: Cannot allocate memory > +++ exited with 1 +++ > > > Thanks, > > Jorge Almeida maybe a memory fragmentation problem? It tries to allocate 65mb continous chunk - and can't find one? -- #163933 ^ permalink raw reply [flat|nested] 24+ messages in thread
* [gentoo-user] Re: xz memory hungry? 2012-08-22 18:52 [gentoo-user] xz memory hungry? Jorge Almeida 2012-08-22 19:05 ` Michael Mol 2012-08-22 20:39 ` Florian Philipp @ 2012-08-23 1:43 ` Nikos Chantziaras 2012-08-23 8:09 ` Jorge Almeida 2 siblings, 1 reply; 24+ messages in thread From: Nikos Chantziaras @ 2012-08-23 1:43 UTC (permalink / raw To: gentoo-user On 22/08/12 21:52, Jorge Almeida wrote: > # tar -xJvf /usr/portage/distfiles/m4-1.4.16.tar.xz > xz: (stdin): Cannot allocate memory > tar: Child returned status 1 > tar: Error is not recoverable: exiting now > > The box has 2G ram + 1G swap. I'm installing Gentoo from an existing distro. > Emerging fails on m4. tar xJvf fails both from within the chroot and from the > host system. top shows that nothing is using any amount of memory worth > mentioning. Extracting libtool-2.4.tar.xz works. I can extract > m4-1.4.16.tar.xz in a computer with 4G ram. This is ridiculous. Not gentoo > related, except that I have no choice, as m4 is pulled by other packages. > What to do? > > app-arch/xz-utils-5.0.3 in chroot > xz 5.0.4 in host system (Archlinux) You should file a bug about this. Whoever put that xz there surely has no idea that this is happening. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [gentoo-user] Re: xz memory hungry? 2012-08-23 1:43 ` [gentoo-user] " Nikos Chantziaras @ 2012-08-23 8:09 ` Jorge Almeida 0 siblings, 0 replies; 24+ messages in thread From: Jorge Almeida @ 2012-08-23 8:09 UTC (permalink / raw To: gentoo-user On Thu, Aug 23, 2012 at 2:43 AM, Nikos Chantziaras <realnc@gmail.com> wrote: > On 22/08/12 21:52, Jorge Almeida wrote: >> > > > You should file a bug about this. Whoever put that xz there surely has no > idea that this is happening. > > Not Gentoo nor xz fault (see other messages in thread). Thanks J.A. ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2012-08-23 11:40 UTC | newest] Thread overview: 24+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-08-22 18:52 [gentoo-user] xz memory hungry? Jorge Almeida 2012-08-22 19:05 ` Michael Mol 2012-08-22 20:12 ` Neil Bothwick 2012-08-22 20:24 ` Michael Mol 2012-08-22 20:32 ` Jorge Almeida 2012-08-22 20:43 ` Florian Philipp 2012-08-22 21:12 ` Michael Mol 2012-08-22 21:10 ` Neil Bothwick 2012-08-22 20:39 ` Florian Philipp 2012-08-22 21:16 ` Jorge Almeida 2012-08-22 21:42 ` Michael Mol 2012-08-22 21:42 ` Michael Mol 2012-08-22 22:19 ` Jorge Almeida 2012-08-23 0:24 ` Jorge Almeida 2012-08-23 0:56 ` Michael Mol 2012-08-23 3:34 ` Paul Hartman 2012-08-23 8:37 ` Jorge Almeida 2012-08-23 9:47 ` Bill Kenworthy 2012-08-23 10:14 ` Jorge Almeida 2012-08-23 11:23 ` Blakawk 2012-08-23 11:37 ` Jorge Almeida 2012-08-23 8:22 ` Volker Armin Hemmann 2012-08-23 1:43 ` [gentoo-user] " Nikos Chantziaras 2012-08-23 8:09 ` Jorge Almeida
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox