* [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
@ 2019-06-13 10:19 Alexey Eschenko
2019-06-13 10:40 ` tedheadster
` (3 more replies)
0 siblings, 4 replies; 19+ messages in thread
From: Alexey Eschenko @ 2019-06-13 10:19 UTC (permalink / raw
To: gentoo-user
[-- Attachment #1: Type: text/html, Size: 1574 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 10:19 [gentoo-user] Not enough RAM for dev-qt/qtwebengine build Alexey Eschenko
@ 2019-06-13 10:40 ` tedheadster
2019-06-13 14:15 ` Alexey Eschenko
2019-06-13 10:50 ` Pouru Lasse
` (2 subsequent siblings)
3 siblings, 1 reply; 19+ messages in thread
From: tedheadster @ 2019-06-13 10:40 UTC (permalink / raw
To: gentoo-user
Alexey,
you can check /etc/portage/make.conf and see if there is a
MAKEOPTS="-j8" or similar variable. If not, add it and make the number
of jobs small, like "-j2". I have a similar problem when I build
sys-devel/binutils; it is a huge memory hog.
- Matthew
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 10:19 [gentoo-user] Not enough RAM for dev-qt/qtwebengine build Alexey Eschenko
2019-06-13 10:40 ` tedheadster
@ 2019-06-13 10:50 ` Pouru Lasse
2019-06-13 11:06 ` John Blinka
2019-06-13 11:13 ` Neil Bothwick
2019-06-13 11:44 ` Kai Peter
3 siblings, 1 reply; 19+ messages in thread
From: Pouru Lasse @ 2019-06-13 10:50 UTC (permalink / raw
To: gentoo-user@lists.gentoo.org
Alexey Eschenko <skobkin-ru@ya.ru> writes:
> Hi.
>
> For some time I have problems with dev-qt/qtwebengine (at least 5.12.3) build. As far as I can see it fails to build due to memory exhaustion. Although I have 32 GB of RAM (at least 20 of them is almost always free) looks like it's not enough. It's strange because I have
> no problems with Firefox/Chromium/Libreoffice builds.
>
> The build ends with full (32GB/32GB) memory usage and significant system lag (mouse/keyboard input, output, video and sound is freezing for ~10+ seconds) every time and then Portage is marking it as failed with something like this in the end of the logs:
> https://cp.skobk.in/2096/ae027ec300f014f6
>
> Also I have something like this in the kernel log:
>
> Jun 13 12:42:24 skobkin-pc kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1.scope,task=cc1plus,pid=39585,uid=250
> Jun 13 12:42:24 skobkin-pc kernel: Out of memory: Killed process 39585 (cc1plus) total-vm:1715764kB, anon-rss:1615872kB, file-rss:0kB, shmem-rss:0kB
> Jun 13 12:42:24 skobkin-pc kernel: oom_reaper: reaped process 39585 (cc1plus), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
>
> It's the first time when I encountered a problem like this on my system and now I'm thinking how to deal with it. Has anyone dealt with this? Is there any solutions other than buying more RAM (I don't need more RAM for my work/entertainment right now)?
>
> Thanks in advance.
I can't really help with the problem, but I've built the same package
recently with just 4GB of RAM. (It takes a long time.) So most likely
it's something in your portage settings that's causing this.
- Lasse
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 10:50 ` Pouru Lasse
@ 2019-06-13 11:06 ` John Blinka
0 siblings, 0 replies; 19+ messages in thread
From: John Blinka @ 2019-06-13 11:06 UTC (permalink / raw
To: gentoo-user
[-- Attachment #1: Type: text/plain, Size: 473 bytes --]
>
> I can't really help with the problem, but I've built the same package
> recently with just 4GB of RAM. (It takes a long time.) So most likely
> it's something in your portage settings that's causing this.
>
> - Lasse
Agreed. I’ve built it recently on 16GB of RAM. My MAKEOPTS is -j13 -l5 to
allow good distribution of effort across 2 other machines with distcc.
Only about 7-8 GB appears to be actively used on the machine running the
emerge.
John
[-- Attachment #2: Type: text/html, Size: 760 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 10:19 [gentoo-user] Not enough RAM for dev-qt/qtwebengine build Alexey Eschenko
2019-06-13 10:40 ` tedheadster
2019-06-13 10:50 ` Pouru Lasse
@ 2019-06-13 11:13 ` Neil Bothwick
2019-06-13 11:48 ` Alexey Eschenko
2019-06-13 11:44 ` Kai Peter
3 siblings, 1 reply; 19+ messages in thread
From: Neil Bothwick @ 2019-06-13 11:13 UTC (permalink / raw
To: gentoo-user
[-- Attachment #1: Type: text/plain, Size: 867 bytes --]
On Thu, 13 Jun 2019 13:19:24 +0300, Alexey Eschenko wrote:
> For some time I have problems with dev-qt/qtwebengine (at least 5.12.3)
> build. As far as I can see it fails to build due to memory exhaustion.
> Although I have 32 GB of RAM (at least 20 of them is almost always
> free) looks like it's not enough. It's strange because I have no
> problems with Firefox/Chromium/Libreoffice builds.
Do you have $PORTAGE_TMPDIR on a tmpfs filesystem? I had a similar
problem, on a slightly less well-endowed system, and had to use
portage.env to instruct that ebuild, as well as chromium and libreoffice,
to use a directory on my SSD for PORTAGE_TMPDIR.
% cat /etc/portage/package.env/qt
dev-qt/qtwebengine disk-tmpdir.conf
% cat /etc/portage/env/disk-tmpdir.conf
PORTAGE_TMPDIR="/mnt/scratch"
--
Neil Bothwick
Assassins do it from behind.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 10:19 [gentoo-user] Not enough RAM for dev-qt/qtwebengine build Alexey Eschenko
` (2 preceding siblings ...)
2019-06-13 11:13 ` Neil Bothwick
@ 2019-06-13 11:44 ` Kai Peter
2019-06-13 11:59 ` Adam Carter
3 siblings, 1 reply; 19+ messages in thread
From: Kai Peter @ 2019-06-13 11:44 UTC (permalink / raw
To: gentoo-user
>
> It's the first time when I encountered a problem like this on my
> system and now I'm thinking how to deal with it. Has anyone dealt with
> this? Is there any solutions other than buying more RAM (I don't need
> more RAM for my work/entertainment right now)?
>
> Thanks in advance.
You can use more swap (files) before buying more RAM.
--
Sent with eQmail-1.10.2 - a fork of djb's famous qmail
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 11:13 ` Neil Bothwick
@ 2019-06-13 11:48 ` Alexey Eschenko
2019-06-13 17:43 ` Neil Bothwick
2019-06-14 6:08 ` J. Roeleveld
0 siblings, 2 replies; 19+ messages in thread
From: Alexey Eschenko @ 2019-06-13 11:48 UTC (permalink / raw
To: gentoo-user@lists.gentoo.org
[-- Attachment #1: Type: text/html, Size: 1559 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 11:44 ` Kai Peter
@ 2019-06-13 11:59 ` Adam Carter
2019-06-13 14:58 ` Mick
0 siblings, 1 reply; 19+ messages in thread
From: Adam Carter @ 2019-06-13 11:59 UTC (permalink / raw
To: gentoo-user
[-- Attachment #1: Type: text/plain, Size: 436 bytes --]
>
> You can use more swap (files) before buying more RAM.
>
>
I have been doing this too. It only get used during the big builds.
To create a 32G swap file and enable it (OP can do this now as the build
runs, to keep OOM away)
# fallocate -l 32G <swapfile> && chmod 600 <swapfile> && mkswap <swapfile>
&& swapon <swapfile>
use 'watch free -m' to see if its being used. Then after the build is over,
delete it/add it to fstab/whatever
[-- Attachment #2: Type: text/html, Size: 793 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 10:40 ` tedheadster
@ 2019-06-13 14:15 ` Alexey Eschenko
2019-06-15 18:18 ` Walter Dnes
0 siblings, 1 reply; 19+ messages in thread
From: Alexey Eschenko @ 2019-06-13 14:15 UTC (permalink / raw
To: gentoo-user@lists.gentoo.org
[-- Attachment #1: Type: text/html, Size: 908 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 11:59 ` Adam Carter
@ 2019-06-13 14:58 ` Mick
2019-06-14 6:32 ` Adam Carter
0 siblings, 1 reply; 19+ messages in thread
From: Mick @ 2019-06-13 14:58 UTC (permalink / raw
To: gentoo-user
[-- Attachment #1: Type: text/plain, Size: 1227 bytes --]
On Thursday, 13 June 2019 12:59:41 BST Adam Carter wrote:
> > You can use more swap (files) before buying more RAM.
>
> I have been doing this too. It only get used during the big builds.
>
> To create a 32G swap file and enable it (OP can do this now as the build
> runs, to keep OOM away)
> # fallocate -l 32G <swapfile> && chmod 600 <swapfile> && mkswap <swapfile>
> && swapon <swapfile>
>
> use 'watch free -m' to see if its being used. Then after the build is over,
> delete it/add it to fstab/whatever
More swap will help, but the basic problem here is the HUGE number of jobs
specified. If each job eats say up to 2G when it gets going, then -j32 will
require >64G RAM to keep running without thrashing the swap device, or running
OOM.
The reasonable thing for big builds is to add an environment variable as
already explained, which restricts the number of jobs. Or for an one off run
something like:
MAKEOPTS="-j8 -l7.8" emerge -1aNDv <package_name>
On a i7 (4 cores x 2 threads) I use -j2 for huge compiles like chromium,
because the PC only has 8G RAM. Smaller packages will emerge with -j12
without breaking into a sweat, although the optimum setting for this PC is "-
j8 -l7.8".
--
Regards,
Mick
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 11:48 ` Alexey Eschenko
@ 2019-06-13 17:43 ` Neil Bothwick
2019-06-13 17:59 ` Dale
2019-06-14 6:08 ` J. Roeleveld
1 sibling, 1 reply; 19+ messages in thread
From: Neil Bothwick @ 2019-06-13 17:43 UTC (permalink / raw
To: gentoo-user
[-- Attachment #1: Type: text/plain, Size: 1831 bytes --]
On Thu, 13 Jun 2019 14:48:04 +0300, Alexey Eschenko wrote:
> That was my first move. Unfortunately it didn't help.
> But in the first answer in this thread I saw a good idea about
> decreasing number of parallel build threads. I have MAKEOPTS="-j32" and
> it's probably the cause of my problem with this package. Right now I'm
> trying separate build environment with decreased job count value and it
> looks promissing. I'll leave the message with results in the reply to
> that message. 13.06.2019, 14:13, "Neil Bothwick" <neil@digimed.co.uk>:
You can set that in package.env too, I do this for Chromium:
% cat /etc/portage/package.env/chromium
www-client/chromium alert-done.conf disk-tmpdir.conf j2.conf
% cat /etc/portage/env/j2.conf
MAKEOPTS="-j2"
> > On Thu, 13 Jun 2019 13:19:24 +0300, Alexey Eschenko wrote:
> > For some time I have problems with dev-qt/qtwebengine (at least
> > 5.12.3)
> >> build. As far as I can see it fails to build due to memory
> >> exhaustion. Although I have 32 GB of RAM (at least 20 of them is
> >> almost always free) looks like it's not enough. It's strange because
> >> I have no problems with Firefox/Chromium/Libreoffice builds.
> >
> >
> > Do you have $PORTAGE_TMPDIR on a tmpfs filesystem? I had a similar
> > problem, on a slightly less well-endowed system, and had to use
> > portage.env to instruct that ebuild, as well as chromium and
> > libreoffice, to use a directory on my SSD for PORTAGE_TMPDIR.
> >
> > % cat /etc/portage/package.env/qt
> > dev-qt/qtwebengine disk-tmpdir.conf
> >
> > % cat /etc/portage/env/disk-tmpdir.conf
> > PORTAGE_TMPDIR="/mnt/scratch"
> >
> > --
> > Neil Bothwick
> >
> >
> > Assassins do it from behind.
--
Neil Bothwick
Always proofread carefully to see if you any words out.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 17:43 ` Neil Bothwick
@ 2019-06-13 17:59 ` Dale
2019-06-13 19:27 ` Neil Bothwick
0 siblings, 1 reply; 19+ messages in thread
From: Dale @ 2019-06-13 17:59 UTC (permalink / raw
To: gentoo-user
Neil Bothwick wrote:
> On Thu, 13 Jun 2019 14:48:04 +0300, Alexey Eschenko wrote:
>
>> That was my first move. Unfortunately it didn't help.
>> But in the first answer in this thread I saw a good idea about
>> decreasing number of parallel build threads. I have MAKEOPTS="-j32" and
>> it's probably the cause of my problem with this package. Right now I'm
>> trying separate build environment with decreased job count value and it
>> looks promissing. I'll leave the message with results in the reply to
>> that message. 13.06.2019, 14:13, "Neil Bothwick" <neil@digimed.co.uk>:
> You can set that in package.env too, I do this for Chromium:
>
> % cat /etc/portage/package.env/chromium
> www-client/chromium alert-done.conf disk-tmpdir.conf j2.conf
>
> % cat /etc/portage/env/j2.conf
> MAKEOPTS="-j2"
>
Has that changed? I have a package.env FILE that contains a list of
packages and the path to a file in /etc/portage/env/ that tells what to
change that for package/ Looks like this:
root@fireball / # cat /etc/portage/package.env/package.env
#www-client/seamonkey ../env/single.conf
#www-client/firefox ../env/single.conf
#www-client/firefox ../env/notmpfs.conf
#www-client/seamonkey ../env/notmpfs.conf
app-office/libreoffice ../env/notmpfs.conf
#sys-devel/gcc ../env/notmpfs.conf
dev-qt/qtwebengine ../env/notmpfs.conf
#dev-qt/qtwebkit ../env/notmpfs.conf
#sci-electronics/kicad ../env/notmpfs.conf
root@fireball / # ls -al /etc/portage/env/
total 16
drwxr-xr-x 2 root root 4096 Sep 3 2017 .
drwxr-xr-x 14 root root 4096 May 31 00:40 ..
-rw-r--r-- 1 root root 35 Sep 3 2017 notmpfs.conf
-rw-r--r-- 1 root root 31 Feb 12 2017 single.conf
root@fireball / # cat /etc/portage/env/notmpfs.conf
PORTAGE_TMPDIR="/var/tmp/notmpfs"
root@fireball / # cat /etc/portage/env/single.conf
#EMERGE_DEFAULT_OPTS="-j1"
root@fireball / #
I think I'm doing this different than what you posted. It works but is
it being changed and I need to update before it stops working?
Dale
:-) :-)
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 17:59 ` Dale
@ 2019-06-13 19:27 ` Neil Bothwick
2019-06-13 20:04 ` Dale
0 siblings, 1 reply; 19+ messages in thread
From: Neil Bothwick @ 2019-06-13 19:27 UTC (permalink / raw
To: gentoo-user
[-- Attachment #1: Type: text/plain, Size: 1930 bytes --]
On Thu, 13 Jun 2019 12:59:18 -0500, Dale wrote:
> Neil Bothwick wrote:
> > You can set that in package.env too, I do this for Chromium:
> >
> > % cat /etc/portage/package.env/chromium
> > www-client/chromium alert-done.conf disk-tmpdir.conf j2.conf
> >
> > % cat /etc/portage/env/j2.conf
> > MAKEOPTS="-j2"
>
> Has that changed? I have a package.env FILE that contains a list of
> packages and the path to a file in /etc/portage/env/ that tells what to
> change that for package/ Looks like this:
>
>
> root@fireball / # cat /etc/portage/package.env/package.env
> #www-client/seamonkey ../env/single.conf
> #www-client/firefox ../env/single.conf
> #www-client/firefox ../env/notmpfs.conf
> #www-client/seamonkey ../env/notmpfs.conf
> app-office/libreoffice ../env/notmpfs.conf
> #sys-devel/gcc ../env/notmpfs.conf
> dev-qt/qtwebengine ../env/notmpfs.conf
> #dev-qt/qtwebkit ../env/notmpfs.conf
> #sci-electronics/kicad ../env/notmpfs.conf
>
> root@fireball / # ls -al /etc/portage/env/
> total 16
> drwxr-xr-x 2 root root 4096 Sep 3 2017 .
> drwxr-xr-x 14 root root 4096 May 31 00:40 ..
> -rw-r--r-- 1 root root 35 Sep 3 2017 notmpfs.conf
> -rw-r--r-- 1 root root 31 Feb 12 2017 single.conf
>
> root@fireball / # cat /etc/portage/env/notmpfs.conf
> PORTAGE_TMPDIR="/var/tmp/notmpfs"
>
> root@fireball / # cat /etc/portage/env/single.conf
> #EMERGE_DEFAULT_OPTS="-j1"
>
> root@fireball / #
>
>
> I think I'm doing this different than what you posted. It works but is
> it being changed and I need to update before it stops working?
The only difference is the way you are specifying the paths. Portage
looks in /etc/portage/env for these anyway, you are telling it to
load /etc/portage/env../env/whatever.conf, which is the same thing.
--
Neil Bothwick
Pound for pound, the amoeba is the most vicious animal on the earth.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 19:27 ` Neil Bothwick
@ 2019-06-13 20:04 ` Dale
0 siblings, 0 replies; 19+ messages in thread
From: Dale @ 2019-06-13 20:04 UTC (permalink / raw
To: gentoo-user
Neil Bothwick wrote:
> On Thu, 13 Jun 2019 12:59:18 -0500, Dale wrote:
>
>> Neil Bothwick wrote:
>>> You can set that in package.env too, I do this for Chromium:
>>>
>>> % cat /etc/portage/package.env/chromium
>>> www-client/chromium alert-done.conf disk-tmpdir.conf j2.conf
>>>
>>> % cat /etc/portage/env/j2.conf
>>> MAKEOPTS="-j2"
>> Has that changed? I have a package.env FILE that contains a list of
>> packages and the path to a file in /etc/portage/env/ that tells what to
>> change that for package/ Looks like this:
>>
>>
>> root@fireball / # cat /etc/portage/package.env/package.env
>> #www-client/seamonkey ../env/single.conf
>> #www-client/firefox ../env/single.conf
>> #www-client/firefox ../env/notmpfs.conf
>> #www-client/seamonkey ../env/notmpfs.conf
>> app-office/libreoffice ../env/notmpfs.conf
>> #sys-devel/gcc ../env/notmpfs.conf
>> dev-qt/qtwebengine ../env/notmpfs.conf
>> #dev-qt/qtwebkit ../env/notmpfs.conf
>> #sci-electronics/kicad ../env/notmpfs.conf
>>
>> root@fireball / # ls -al /etc/portage/env/
>> total 16
>> drwxr-xr-x 2 root root 4096 Sep 3 2017 .
>> drwxr-xr-x 14 root root 4096 May 31 00:40 ..
>> -rw-r--r-- 1 root root 35 Sep 3 2017 notmpfs.conf
>> -rw-r--r-- 1 root root 31 Feb 12 2017 single.conf
>>
>> root@fireball / # cat /etc/portage/env/notmpfs.conf
>> PORTAGE_TMPDIR="/var/tmp/notmpfs"
>>
>> root@fireball / # cat /etc/portage/env/single.conf
>> #EMERGE_DEFAULT_OPTS="-j1"
>>
>> root@fireball / #
>>
>>
>> I think I'm doing this different than what you posted. It works but is
>> it being changed and I need to update before it stops working?
> The only difference is the way you are specifying the paths. Portage
> looks in /etc/portage/env for these anyway, you are telling it to
> load /etc/portage/env../env/whatever.conf, which is the same thing.
>
>
OK. If something was changing, I wanted to update if needed. I sort of
think your way is simpler tho. It took me a while to figure out how to
make mine work. At the time, I couldn't find a lot of examples.
Thanks for the info.
Dale
:-) :-)
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 11:48 ` Alexey Eschenko
2019-06-13 17:43 ` Neil Bothwick
@ 2019-06-14 6:08 ` J. Roeleveld
1 sibling, 0 replies; 19+ messages in thread
From: J. Roeleveld @ 2019-06-14 6:08 UTC (permalink / raw
To: gentoo-user
On June 13, 2019 11:48:04 AM UTC, Alexey Eschenko <skobkin-ru@ya.ru> wrote:
>That was my first move. Unfortunately it didn't help.
>
>But in the first answer in this thread I saw a good idea about
>decreasing number of parallel build threads.
>
>I have MAKEOPTS="-j32" and it's probably the cause of my problem with
>this package.
>
>Right now I'm trying separate build environment with decreased job
>count value and it looks promissing.
>
>I'll leave the message with results in the reply to that message.
>
When using --jobs (I had issues with -j once because a different build tool used it for something else) it is a good idea to also add --load-average. This will limit the amount of jobs when the load average goes up. (And that goes up fast when swap is touched)
--
Joost
>
>13.06.2019, 14:13, "Neil Bothwick" <neil@digimed.co.uk>:
>
>On Thu, 13 Jun 2019 13:19:24 +0300, Alexey Eschenko wrote:
>
>
>For some time I have problems with dev-qt/qtwebengine (at least 5.12.3)
> build. As far as I can see it fails to build due to memory exhaustion.
> Although I have 32 GB of RAM (at least 20 of them is almost always
> free) looks like it's not enough. It's strange because I have no
> problems with Firefox/Chromium/Libreoffice builds.
>
>
>Do you have $PORTAGE_TMPDIR on a tmpfs filesystem? I had a similar
>problem, on a slightly less well-endowed system, and had to use
>portage.env to instruct that ebuild, as well as chromium and
>libreoffice,
>to use a directory on my SSD for PORTAGE_TMPDIR.
>
>% cat /etc/portage/package.env/qt
>dev-qt/qtwebengine disk-tmpdir.conf
>
>% cat /etc/portage/env/disk-tmpdir.conf
>PORTAGE_TMPDIR="/mnt/scratch"
>
>
>
>--
>Neil Bothwick
>
>
>Assassins do it from behind.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 14:58 ` Mick
@ 2019-06-14 6:32 ` Adam Carter
0 siblings, 0 replies; 19+ messages in thread
From: Adam Carter @ 2019-06-14 6:32 UTC (permalink / raw
To: gentoo-user
[-- Attachment #1: Type: text/plain, Size: 464 bytes --]
>
> More swap will help, but the basic problem here is the HUGE number of jobs
> specified. If each job eats say up to 2G when it gets going, then -j32
> will
> require >64G RAM to keep running without thrashing the swap device, or
> running
> OOM.
>
Yes agree. If the swapfile is on an SSD then there's no thrashing. For me
the swap file only gets used on big builds and /proc/sys/vm/swappiness is
tweaked via /etc/sysctl.conf so SSD wear is not a big concern.
[-- Attachment #2: Type: text/html, Size: 695 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-13 14:15 ` Alexey Eschenko
@ 2019-06-15 18:18 ` Walter Dnes
2019-06-15 18:37 ` Alec Ten Harmsel
0 siblings, 1 reply; 19+ messages in thread
From: Walter Dnes @ 2019-06-15 18:18 UTC (permalink / raw
To: gentoo-user
On Thu, Jun 13, 2019 at 05:15:41PM +0300, Alexey Eschenko wrote
> Thank you. Didn't think about that. Don't know why though. My
> MAKEOPTS was "-j32". Looks like that was too many for package like
> qtwebengine. Solved the problem with creating specific environment
> for qtwebengine and setting it up in /etc/portage/package.env/ It was
> veeeery long build process but this time it finished successfully. I
> think I'll try to find more appropriate value for qtwebengine which
> could be used with 32GB of RAM.
Please use plain text, not HTML.
According to
https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/
the fastest compiles come with setting MAKEOPTS to the number of cores
in your machine. E.g. for a dual core cpu, use "-j2", for a 4 core cpu
use "-j4", etc. To check the number of cpus in your machine, execute...
grep -c ^flags /proc/cpuinfo
--
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-15 18:18 ` Walter Dnes
@ 2019-06-15 18:37 ` Alec Ten Harmsel
2019-06-15 20:02 ` Dale
0 siblings, 1 reply; 19+ messages in thread
From: Alec Ten Harmsel @ 2019-06-15 18:37 UTC (permalink / raw
To: gentoo-user
On Sat, Jun 15, 2019, at 14:19, Walter Dnes wrote:
> On Thu, Jun 13, 2019 at 05:15:41PM +0300, Alexey Eschenko wrote
> > Thank you. Didn't think about that. Don't know why though. My
> > MAKEOPTS was "-j32". Looks like that was too many for package like
> > qtwebengine. Solved the problem with creating specific environment
> > for qtwebengine and setting it up in /etc/portage/package.env/ It was
> > veeeery long build process but this time it finished successfully. I
> > think I'll try to find more appropriate value for qtwebengine which
> > could be used with 32GB of RAM.
>
> Please use plain text, not HTML.
>
> According to
> https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/
> the fastest compiles come with setting MAKEOPTS to the number of cores
> in your machine. E.g. for a dual core cpu, use "-j2", for a 4 core cpu
> use "-j4", etc. To check the number of cpus in your machine, execute...
>
> grep -c ^flags /proc/cpuinfo
That's a good rule but not necessarily always true. My old machine was an i7-3930K (6 cores, 12 threads) w/ 32G RAM. I had /var/tmp on tmpfs. I benchmarked firefox, chromium, and some other big projects once and -j13 was consistently the fastest on that box.
As that blog post says:
> I’m just saying, ${core} + 1 is not the best optimization for me
> and the test confirms the part:“but this guideline isn’t always perfect”
Depends on available RAM, how fast your disk is, etc.
Alec
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [gentoo-user] Not enough RAM for dev-qt/qtwebengine build
2019-06-15 18:37 ` Alec Ten Harmsel
@ 2019-06-15 20:02 ` Dale
0 siblings, 0 replies; 19+ messages in thread
From: Dale @ 2019-06-15 20:02 UTC (permalink / raw
To: gentoo-user
Alec Ten Harmsel wrote:
> On Sat, Jun 15, 2019, at 14:19, Walter Dnes wrote:
>> On Thu, Jun 13, 2019 at 05:15:41PM +0300, Alexey Eschenko wrote
>>> Thank you. Didn't think about that. Don't know why though. My
>>> MAKEOPTS was "-j32". Looks like that was too many for package like
>>> qtwebengine. Solved the problem with creating specific environment
>>> for qtwebengine and setting it up in /etc/portage/package.env/ It was
>>> veeeery long build process but this time it finished successfully. I
>>> think I'll try to find more appropriate value for qtwebengine which
>>> could be used with 32GB of RAM.
>> Please use plain text, not HTML.
>>
>> According to
>> https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/
>> the fastest compiles come with setting MAKEOPTS to the number of cores
>> in your machine. E.g. for a dual core cpu, use "-j2", for a 4 core cpu
>> use "-j4", etc. To check the number of cpus in your machine, execute...
>>
>> grep -c ^flags /proc/cpuinfo
> That's a good rule but not necessarily always true. My old machine was an i7-3930K (6 cores, 12 threads) w/ 32G RAM. I had /var/tmp on tmpfs. I benchmarked firefox, chromium, and some other big projects once and -j13 was consistently the fastest on that box.
>
> As that blog post says:
>
>> I’m just saying, ${core} + 1 is not the best optimization for me
>> and the test confirms the part:“but this guideline isn’t always perfect”
> Depends on available RAM, how fast your disk is, etc.
>
> Alec
>
>
As a AMD CPU user, I always set mine to number of cores plus one. That
seems to always be the most efficient for me. I have portages work
directory on tmpfs for all but a couple large packages. I used to make
exceptions for Firefox, Seamonkey, Libreoffice and qtweb something or
other. Since I upgraded my memory to 32GBs I removed all but
Libreoffice. My biggest problem was when more than one of those wanted
to compile at the same time.
The best way to know what to set it to, test it. Set it to cores plus
one and test. Then set it to half the number of cores, twice the number
of cores etc until you find that sweet spot. One could even do that
during normal updates although it may not be as accurate. I suspect if
ten people with ten different systems tested this, we'd get half a dozen
different results, maybe more.
The best thing in my opinion, start emerge, go to bed and then hope it
is done when you wake up. lol
Dale
:-) :-)
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2019-06-15 20:02 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-06-13 10:19 [gentoo-user] Not enough RAM for dev-qt/qtwebengine build Alexey Eschenko
2019-06-13 10:40 ` tedheadster
2019-06-13 14:15 ` Alexey Eschenko
2019-06-15 18:18 ` Walter Dnes
2019-06-15 18:37 ` Alec Ten Harmsel
2019-06-15 20:02 ` Dale
2019-06-13 10:50 ` Pouru Lasse
2019-06-13 11:06 ` John Blinka
2019-06-13 11:13 ` Neil Bothwick
2019-06-13 11:48 ` Alexey Eschenko
2019-06-13 17:43 ` Neil Bothwick
2019-06-13 17:59 ` Dale
2019-06-13 19:27 ` Neil Bothwick
2019-06-13 20:04 ` Dale
2019-06-14 6:08 ` J. Roeleveld
2019-06-13 11:44 ` Kai Peter
2019-06-13 11:59 ` Adam Carter
2019-06-13 14:58 ` Mick
2019-06-14 6:32 ` Adam Carter
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox