[gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Sam James]

Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
amount of RAM available (uses amount declared as needed
in the ebuild). Typically should be ~2GB per job.

Bug: https://bugs.gentoo.org/570534
Signed-off-by: Sam James <sam@gentoo.org>
---
 eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 39 insertions(+), 3 deletions(-)

diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass
index 2130e2e349141..8c4adc8b4f121 100644
--- a/eclass/check-reqs.eclass
+++ b/eclass/check-reqs.eclass
@@ -43,6 +43,8 @@ case ${EAPI} in
 	*) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;;
 esac
 
+inherit multiprocessing
+
 EXPORT_FUNCTIONS pkg_pretend pkg_setup
 
 if [[ ! ${_CHECK_REQS_ECLASS} ]]; then
@@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1
 # @DESCRIPTION:
 # How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M
 
+# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
+# @USER_VARIABLE
+# @DESCRIPTION:
+# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
+# to lower memory usage.
+: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
+
 # @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD
 # @DEFAULT_UNSET
 # @DESCRIPTION:
@@ -346,9 +355,36 @@ _check-reqs_memory() {
 			eend 0
 		else
 			eend 1
-			_check-reqs_unsatisfied \
-				${size} \
-				"RAM"
+
+			# Has the user allowed us to mangle their MAKEOPTS?
+			if [[ ${CHECKREQS_MEMORY_MANGLE_JOBS} == "yes" ]] ; then
+				local jobs=$(makeopts_jobs)
+
+				local estimated_max_memory=$((${actual_memory}/$(_check-reqs_get_kibibytes 1G)))
+				if [[ $((jobs*2)) -gt ${estimated_max_memory} ]] ; then
+					# Number of jobs exceeds RAM/2GB, so clamp it.
+					local new_jobs=$(($(_check-reqs_get_number ${estimated_max_memory}G)*10/20))
+
+					# This might _still_ be too big on small machines. Give up in such cases.
+					# (Users can still set the do nothing variable which is independent of this.)
+					if [[ $((new_jobs*2)) -gt ${estimated_max_memory} ]] ; then
+						_check-reqs_unsatisfied \
+							${size} \
+							"RAM"
+					else
+						# The clamped jobs seem to be enough to satisfy the check-reqs requirement from the ebuild.
+						ewarn "Clamping MAKEOPTS jobs to -j${new_jobs} to reduce memory usage"
+						ewarn "Compiler jobs may use around ~2GB each: https://wiki.gentoo.org/wiki/MAKEOPTS"
+						ewarn "To disable this, set CHECKREQS_MEMORY_MANGLE_JOBS=no."
+
+						MAKEOPTS+=" -j${new_jobs}"
+					fi
+				fi
+			else
+				_check-reqs_unsatisfied \
+					${size} \
+					"RAM"
+			fi
 		fi
 	else
 		eend 1
-- 
2.34.1

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Sam James]

[-- Attachment #1.1: Type: text/plain, Size: 892 bytes --]



> On 4 Jan 2022, at 22:58, Sam James <sam@gentoo.org> wrote:
> 
> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> amount of RAM available (uses amount declared as needed
> in the ebuild). Typically should be ~2GB per job.
> 
> Bug: https://bugs.gentoo.org/570534
> Signed-off-by: Sam James <sam@gentoo.org>
> ---
> eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
> 1 file changed, 39 insertions(+), 3 deletions(-)

Note that we discussed this on GitHub a bit when I just posted it there
for some rough feedback: https://github.com/gentoo/gentoo/pull/23311 <https://github.com/gentoo/gentoo/pull/23311>.

I think this is valuable for reducing invalid bug reports from OOM and
easing user experience.

Still kind of a WIP/rough draft, but may be ready in this state. Need
more testing, so not planning on pushing yet or anything.

[-- Attachment #1.2: Type: text/html, Size: 1713 bytes --]

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Ulrich Mueller]

>>>>> On Tue, 04 Jan 2022, Sam James wrote:

> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> amount of RAM available (uses amount declared as needed
> in the ebuild). Typically should be ~2GB per job.

Where does this number 2 GB come from? The amount of RAM strongly
depends on the programming language and other factors, so I don't
believe that there's one number that can be used for everything.
(If only considering C and C++ 2 GB seems to be excessive, i.e. it
will limit parallel build more than necessary. When linking, the number
may be _much_ larger.)

Also not sure if I understand the arithmetic. Shouldn't it use
CHECKREQS_MEMORY as the basis of the calculation?

> +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
> +# @USER_VARIABLE
> +# @DESCRIPTION:
> +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
> +# to lower memory usage.
> +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}

If anything, the feature should be opt-in rather than opt-out.

Ulrich

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Florian Schmaus]

On 05/01/2022 09.28, Ulrich Mueller wrote:
>>>>>> On Tue, 04 Jan 2022, Sam James wrote:
>> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
>> amount of RAM available (uses amount declared as needed
>> in the ebuild). Typically should be ~2GB per job.
> 
> Where does this number 2 GB come from? The amount of RAM strongly
> depends on the programming language and other factors, so I don't
> believe that there's one number that can be used for everything.

Surely not, but 2 GiB seems like a good start. I guess it could become 
an eclass variable that ebuilds could modify later (if the need emerges).


>> +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
>> +# @USER_VARIABLE
>> +# @DESCRIPTION:
>> +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
>> +# to lower memory usage.
>> +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
> 
> If anything, the feature should be opt-in rather than opt-out.

It appears to me that the motivation for this change is to prevent users 
from running into OOM situations when emerging a package. And I believe 
that many users, especially novices, are not directly able to determine 
that portage failed due to OOM, simply because there is no direct hint 
in the build log. Hence opt-out appears to be sensible to me here.

- Flow

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Alexey Sokolov]

I do sometimes run into OOM during emerge, but for a different reason: 
when firefox and webengine are built in parallel. And I'm using tmpfs 
for portage. These packages store lots of data in the build directory. 
Decreasing -j of a single package may or may not help in this case.

The filled tmpfs issue becomes more severe if I have tests enabled, 
because ctest has different behavior from make/ninja, and I use -j with 
combination with -l: make and ninja run at least one job to ensure some 
progress, while ctest does nothing when the load average is too high, 
and while it's doing nothing, the portage cannot finish the build and 
therefore cleanup the build dir from RAM.

Neither of these are exactly relevant to this patch, tbh.

04.01.2022 22:58, Sam James пишет:
> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> amount of RAM available (uses amount declared as needed
> in the ebuild). Typically should be ~2GB per job.
> 
> Bug: https://bugs.gentoo.org/570534
> Signed-off-by: Sam James <sam@gentoo.org>
> ---
>   eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
>   1 file changed, 39 insertions(+), 3 deletions(-)
> 
> diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass
> index 2130e2e349141..8c4adc8b4f121 100644
> --- a/eclass/check-reqs.eclass
> +++ b/eclass/check-reqs.eclass
> @@ -43,6 +43,8 @@ case ${EAPI} in
>   	*) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;;
>   esac
>   
> +inherit multiprocessing
> +
>   EXPORT_FUNCTIONS pkg_pretend pkg_setup
>   
>   if [[ ! ${_CHECK_REQS_ECLASS} ]]; then
> @@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1
>   # @DESCRIPTION:
>   # How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M
>   
> +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
> +# @USER_VARIABLE
> +# @DESCRIPTION:
> +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
> +# to lower memory usage.
> +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
> +
>   # @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD
>   # @DEFAULT_UNSET
>   # @DESCRIPTION:
> @@ -346,9 +355,36 @@ _check-reqs_memory() {
>   			eend 0
>   		else
>   			eend 1
> -			_check-reqs_unsatisfied \
> -				${size} \
> -				"RAM"
> +
> +			# Has the user allowed us to mangle their MAKEOPTS?
> +			if [[ ${CHECKREQS_MEMORY_MANGLE_JOBS} == "yes" ]] ; then
> +				local jobs=$(makeopts_jobs)
> +
> +				local estimated_max_memory=$((${actual_memory}/$(_check-reqs_get_kibibytes 1G)))
> +				if [[ $((jobs*2)) -gt ${estimated_max_memory} ]] ; then
> +					# Number of jobs exceeds RAM/2GB, so clamp it.
> +					local new_jobs=$(($(_check-reqs_get_number ${estimated_max_memory}G)*10/20))
> +
> +					# This might _still_ be too big on small machines. Give up in such cases.
> +					# (Users can still set the do nothing variable which is independent of this.)
> +					if [[ $((new_jobs*2)) -gt ${estimated_max_memory} ]] ; then
> +						_check-reqs_unsatisfied \
> +							${size} \
> +							"RAM"
> +					else
> +						# The clamped jobs seem to be enough to satisfy the check-reqs requirement from the ebuild.
> +						ewarn "Clamping MAKEOPTS jobs to -j${new_jobs} to reduce memory usage"
> +						ewarn "Compiler jobs may use around ~2GB each: https://wiki.gentoo.org/wiki/MAKEOPTS"
> +						ewarn "To disable this, set CHECKREQS_MEMORY_MANGLE_JOBS=no."
> +
> +						MAKEOPTS+=" -j${new_jobs}"
> +					fi
> +				fi
> +			else
> +				_check-reqs_unsatisfied \
> +					${size} \
> +					"RAM"
> +			fi
>   		fi
>   	else
>   		eend 1


-- 
Best regards,
Alexey "DarthGandalf" Sokolov

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Alec Warner]

On Tue, Jan 4, 2022 at 3:03 PM Sam James <sam@gentoo.org> wrote:
>
>
>
> On 4 Jan 2022, at 22:58, Sam James <sam@gentoo.org> wrote:
>
> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> amount of RAM available (uses amount declared as needed
> in the ebuild). Typically should be ~2GB per job.
>
> Bug: https://bugs.gentoo.org/570534
> Signed-off-by: Sam James <sam@gentoo.org>
> ---
> eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
> 1 file changed, 39 insertions(+), 3 deletions(-)
>
>
> Note that we discussed this on GitHub a bit when I just posted it there
> for some rough feedback: https://github.com/gentoo/gentoo/pull/23311.
>
> I think this is valuable for reducing invalid bug reports from OOM and
> easing user experience.
>
> Still kind of a WIP/rough draft, but may be ready in this state. Need
> more testing, so not planning on pushing yet or anything.

I'm still not sure I grasp why we cannot make OOMs easier to discover
from portage.

Most packages don't even use check-reqs, so your solution is very
narrow (and I get why, because you get a lot of bug reports from the
big packages that do use it.)

Can we write a build log analyzer?

-A

PS: If this was a global change I'd downvote it. It's only for
check-reqs though and most packages don't use check-reqs; I don't
really care. I'd be concerned about adopting this kind of approach
wider; its very much a bandaid.

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

>>>>> On Wed, 05 Jan 2022, Florian Schmaus wrote:

> On 05/01/2022 09.28, Ulrich Mueller wrote:
>>>>>>> On Tue, 04 Jan 2022, Sam James wrote:
>>> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
>>> amount of RAM available (uses amount declared as needed
>>> in the ebuild). Typically should be ~2GB per job.
>> Where does this number 2 GB come from? The amount of RAM strongly
>> depends on the programming language and other factors, so I don't
>> believe that there's one number that can be used for everything.

> Surely not, but 2 GiB seems like a good start. I guess it could become
> an eclass variable that ebuilds could modify later (if the need
> emerges).

We already have an eclass variable, namely CHECKREQS_MEMORY. Do you want
to introduce another one that is counting memory per job? I doubt that
would be possible, because the amount of memory greatly varies within a
single build. For example, it is different for compiler vs linker vs
building the documentation.

> It appears to me that the motivation for this change is to prevent
> users from running into OOM situations when emerging a package. And I
> believe that many users, especially novices, are not directly able to
> determine that portage failed due to OOM, simply because there is no
> direct hint in the build log. Hence opt-out appears to be sensible to
> me here.

That applies to all parallel builds though, not only to ebuilds
inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
telling the user that the --jobs setting in their make.conf is wrong,
in the first place.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 3075 bytes --]

On 2022.01.04 22:58, Sam James wrote:
> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> amount of RAM available (uses amount declared as needed
> in the ebuild). Typically should be ~2GB per job.
> 
> Bug: https://bugs.gentoo.org/570534
> Signed-off-by: Sam James <sam@gentoo.org>
> ---
>  eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++---
>  1 file changed, 39 insertions(+), 3 deletions(-)
> 
> diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass
> index 2130e2e349141..8c4adc8b4f121 100644
> --- a/eclass/check-reqs.eclass
> +++ b/eclass/check-reqs.eclass
> @@ -43,6 +43,8 @@ case ${EAPI} in
>  	*) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;;
>  esac
>  
> +inherit multiprocessing
> +
>  EXPORT_FUNCTIONS pkg_pretend pkg_setup
>  
>  if [[ ! ${_CHECK_REQS_ECLASS} ]]; then
> @@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1
>  # @DESCRIPTION:
>  # How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M
>  
> +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
> +# @USER_VARIABLE
> +# @DESCRIPTION:
> +# Allow packages to reduce the number of multiprocessing (e.g. make,
> ninja) jobs
> +# to lower memory usage.
> +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
> +
>  # @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD
>  # @DEFAULT_UNSET
>  # @DESCRIPTION:
> @@ -346,9 +355,36 @@ _check-reqs_memory() {
>  			eend 0
>  		else
>  			eend 1
> -			_check-reqs_unsatisfied \
> -				${size} \
> -				"RAM"
> +
> +			# Has the user allowed us to mangle their
> MAKEOPTS?
> +			if [[ ${CHECKREQS_MEMORY_MANGLE_JOBS} ==
> "yes" ]] ; then
> +				local jobs=$(makeopts_jobs)
> +
> +				local estimated_max_memory=$((${actual_memory}/$(_check-reqs_get_kibibytes
> 1G)))
> +				if [[ $((jobs*2)) -gt ${estimated_max_memory}
> ]] ; then
> +					# Number of jobs exceeds RAM/2GB,
> so clamp it.
> +					local new_jobs=$(($(_check-reqs_get_number
> ${estimated_max_memory}G)*10/20))
> +
> +					# This might _still_ be too big
> on small machines. Give up in such cases.
> +					# (Users can still set the do
> nothing variable which is independent of this.)
> +					if [[ $((new_jobs*2)) -gt ${estimated_max_memory}
> ]] ; then
> +						_check-reqs_unsatisfied \
> +							${size} \
> +							"RAM"
> +					else
> +						# The clamped jobs seem to be
> enough to satisfy the check-reqs requirement from the ebuild.
> +						ewarn "Clamping MAKEOPTS jobs
> to -j${new_jobs} to reduce memory usage"
> +						ewarn "Compiler jobs may use
> around ~2GB each: https://wiki.gentoo.org/wiki/MAKEOPTS"
> +						ewarn "To disable this, set CHECKREQS_MEMORY_MANGLE_JOBS=no."
> +
> +						MAKEOPTS+=" -j${new_jobs}"
> +					fi
> +				fi
> +			else
> +				_check-reqs_unsatisfied \
> +					${size} \
> +					"RAM"
> +			fi
>  		fi
>  	else
>  		eend 1
> -- 
> 2.34.1
> 
> 
> 
> 

Sam,

Do users with FEATURES=distcc still have to opt out of this 
MAKEOPTS clamping?

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
arm64

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Florian Schmaus]

On 05/01/2022 19.22, Ulrich Mueller wrote:
>>>>>> On Wed, 05 Jan 2022, Florian Schmaus wrote:
> 
>> On 05/01/2022 09.28, Ulrich Mueller wrote:
>>>>>>>> On Tue, 04 Jan 2022, Sam James wrote:
>>>> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
>>>> amount of RAM available (uses amount declared as needed
>>>> in the ebuild). Typically should be ~2GB per job.
>>> Where does this number 2 GB come from? The amount of RAM strongly
>>> depends on the programming language and other factors, so I don't
>>> believe that there's one number that can be used for everything.
> 
>> Surely not, but 2 GiB seems like a good start. I guess it could become
>> an eclass variable that ebuilds could modify later (if the need
>> emerges).
> 
> We already have an eclass variable, namely CHECKREQS_MEMORY. Do you want
> to introduce another one that is counting memory per job? I doubt that
> would be possible, because the amount of memory greatly varies within a
> single build. For example, it is different for compiler vs linker vs
> building the documentation.

I am not sure what the problem here supposed to be is.


>> It appears to me that the motivation for this change is to prevent
>> users from running into OOM situations when emerging a package. And I
>> believe that many users, especially novices, are not directly able to
>> determine that portage failed due to OOM, simply because there is no
>> direct hint in the build log. Hence opt-out appears to be sensible to
>> me here.
> 
> That applies to all parallel builds though, not only to ebuilds
> inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
> telling the user that the --jobs setting in their make.conf is wrong,
> in the first place.

Yes, exactly. And it is a bandaid solution. But I believe that it will 
do more good than evil. And it is probably only used until portage is 
able to report to the user that the emerge failed due to OOM (which I 
believe to be non-trivial to implement, but I am happy to be proven 
otherwise).

- Flow

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Kai Krakow]

Am Mi., 5. Jan. 2022 um 19:22 Uhr schrieb Ulrich Mueller <ulm@gentoo.org>:
>
> >>>>> On Wed, 05 Jan 2022, Florian Schmaus wrote:
>
> > On 05/01/2022 09.28, Ulrich Mueller wrote:
> >>>>>>> On Tue, 04 Jan 2022, Sam James wrote:
> >>> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> >>> amount of RAM available (uses amount declared as needed
> >>> in the ebuild). Typically should be ~2GB per job.
> >> Where does this number 2 GB come from? The amount of RAM strongly
> >> depends on the programming language and other factors, so I don't
> >> believe that there's one number that can be used for everything.
>
> > Surely not, but 2 GiB seems like a good start. I guess it could become
> > an eclass variable that ebuilds could modify later (if the need
> > emerges).
>
> We already have an eclass variable, namely CHECKREQS_MEMORY. Do you want
> to introduce another one that is counting memory per job? I doubt that
> would be possible, because the amount of memory greatly varies within a
> single build. For example, it is different for compiler vs linker vs
> building the documentation.
>
> > It appears to me that the motivation for this change is to prevent
> > users from running into OOM situations when emerging a package. And I
> > believe that many users, especially novices, are not directly able to
> > determine that portage failed due to OOM, simply because there is no
> > direct hint in the build log. Hence opt-out appears to be sensible to
> > me here.
>
> That applies to all parallel builds though, not only to ebuilds
> inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
> telling the user that the --jobs setting in their make.conf is wrong,
> in the first place.

Well, I'm using a safe combination of jobs and load-average, maybe the
documentation should be tweaked instead.

I'm using

MAkEOPTS="--jobs=12 --load-average=8"

on a 8x2+4 core CPU. I choose less then the available 20 cores because
I'm also running multiple ebuilds in parallel to make better use of
resources while most packages run their configure phase (which for
small packages always takes longer than the build itself):

EMERGE_DEFAULT_OPTS="--jobs=12 --load-average 10"

The "--jobs" parameter is mostly a safe-guard against "make" or
"emerge" overshooting the system resources which would happen if
running unconstrained without "--load-average". The latter parameter
OTOH tunes the parallel building processes automatically to the
available resources. If the system starves of memory, thus starts to
swap, load will increase, and make will reduce the jobs. It works
pretty well.

I've chosen the emerge loadavg limit slightly higher so a heavy ebuild
won't starve emerge from running configure phases of parallel ebuilds.

Regards,
Kai

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1045 bytes --]

>>>>> On Wed, 05 Jan 2022, Florian Schmaus wrote:

>> That applies to all parallel builds though, not only to ebuilds
>> inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
>> telling the user that the --jobs setting in their make.conf is wrong,
>> in the first place.

> Yes, exactly. And it is a bandaid solution. But I believe that it will
> do more good than evil. And it is probably only used until portage is 
> able to report to the user that the emerge failed due to OOM (which I
> believe to be non-trivial to implement, but I am happy to be proven 
> otherwise).

Obviously I disagree. Tweaking the value for a subset of packages isn't
a solution to the problem.

MAKEOPTS applies to all parallel builds, and users should set it to a
value suitable for their system (i.e. number of CPUs, available memory,
etc.). Maybe our documentation needs to be improved? I see that
make.conf.example says "The suggested number for parallel makes is
CPUs+1" which may not be the best possible advice.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1540 bytes --]



> On 5 Jan 2022, at 19:53, Ulrich Mueller <ulm@gentoo.org> wrote:
> 
>>>>>> On Wed, 05 Jan 2022, Florian Schmaus wrote:
> 
>>> That applies to all parallel builds though, not only to ebuilds
>>> inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
>>> telling the user that the --jobs setting in their make.conf is wrong,
>>> in the first place.
> 
>> Yes, exactly. And it is a bandaid solution. But I believe that it will
>> do more good than evil. And it is probably only used until portage is
>> able to report to the user that the emerge failed due to OOM (which I
>> believe to be non-trivial to implement, but I am happy to be proven
>> otherwise).
> 
> Obviously I disagree. Tweaking the value for a subset of packages isn't
> a solution to the problem.
> 
> MAKEOPTS applies to all parallel builds, and users should set it to a
> value suitable for their system (i.e. number of CPUs, available memory,
> etc.). Maybe our documentation needs to be improved? I see that
> make.conf.example says "The suggested number for parallel makes is
> CPUs+1" which may not be the best possible advice.
> 

As you've noted, the memory requirements vary per package,
and it was somewhat of an oversight/error to not include
the memory limit from the eclass variable instead here
(although I recall kind of deciding to go with this approach
for simplicity).

It's therefore nontrivial to come up with a good value for their
whole system :)

Our documentation has mostly been updated but apparently
make.conf.example hasn't been.


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 2202 bytes --]



> On 5 Jan 2022, at 08:28, Ulrich Mueller <ulm@gentoo.org> wrote:
> 
>>>>>> On Tue, 04 Jan 2022, Sam James wrote:
> 
>> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
>> amount of RAM available (uses amount declared as needed
>> in the ebuild). Typically should be ~2GB per job.
> 
> Where does this number 2 GB come from? The amount of RAM strongly
> depends on the programming language and other factors, so I don't
> believe that there's one number that can be used for everything.
> (If only considering C and C++ 2 GB seems to be excessive, i.e. it
> will limit parallel build more than necessary. When linking, the number
> may be _much_ larger.)
> 

This is essentially "common law" (or "common lore" if you like!) and
is well-accepted as a good rule of thumb.

The number being larger doesn't really make any difference here;
the point is to help in cases where we're pretty sure there's going
to be a problem (only users of check-reqs.eclass, and we can
Introduce a variable to give ~RAM per job too if needed).

> Also not sure if I understand the arithmetic. Shouldn't it use
> CHECKREQS_MEMORY as the basis of the calculation?

As noted, I was sort of on the fence about whether to hardcode
or use the CHECKREQS_MEMORY value and I'll think about this
again. I went for the current approach partly to keep things simple
(to avoid needing min/max bits).

> 
>> +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
>> +# @USER_VARIABLE
>> +# @DESCRIPTION:
>> +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs
>> +# to lower memory usage.
>> +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
> 
> If anything, the feature should be opt-in rather than opt-out.

That would defeat the purpose. If it's opt-in, then people are already aware
of their settings being inappropriate, and should probably fix them.

This is also about reducing the number of support queries about
builds which failed for trivial reasons, as someone who has to handle
quite a lot of those (until such a time as we can implement better
detection, but this is also a generally nice UX improvement -- as
per what Florian/flow said).


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Sam James]

[-- Attachment #1.1: Type: text/plain, Size: 1518 bytes --]



> On 5 Jan 2022, at 19:18, Kai Krakow <kai@kaishome.de> wrote:
> 
> Am Mi., 5. Jan. 2022 um 19:22 Uhr schrieb Ulrich Mueller <ulm@gentoo.org <mailto:ulm@gentoo.org>>:
>> 
>>>>>>> [...]
>> That applies to all parallel builds though, not only to ebuilds
>> inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
>> telling the user that the --jobs setting in their make.conf is wrong,
>> in the first place.
> 
> Well, I'm using a safe combination of jobs and load-average, maybe the
> documentation should be tweaked instead.
> 

I think "safe" is doing some heavy lifting here...

> I'm using
> [...]

> 
> The "--jobs" parameter is mostly a safe-guard against "make" or
> "emerge" overshooting the system resources which would happen if
> running unconstrained without "--load-average". The latter parameter
> OTOH tunes the parallel building processes automatically to the
> available resources. If the system starves of memory, thus starts to
> swap, load will increase, and make will reduce the jobs. It works
> pretty well.
> 
> I've chosen the emerge loadavg limit slightly higher so a heavy ebuild
> won't starve emerge from running configure phases of parallel ebuilds.
> 

... because it's quite hard for this logic to work correctly enough
of the time without jobserver integration (https://bugs.gentoo.org/692576 <https://bugs.gentoo.org/692576>).

But indeed, I'd say you're not the target audience for this (but I appreciate
the input).

Best,
sam

[-- Attachment #1.2: Type: text/html, Size: 15083 bytes --]

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]



> On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org> wrote:
> 
> Sam,
> 
> Do users with FEATURES=distcc still have to opt out of this
> MAKEOPTS clamping?
> 

Great point! I think we could add an exemption for that and make it a noop or warning-only.

Best,
sam


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]



> On 5 Jan 2022, at 17:51, Alec Warner <antarus@gentoo.org> wrote:
> On Tue, Jan 4, 2022 at 3:03 PM Sam James <sam@gentoo.org> wrote:
>> 
>> On 4 Jan 2022, at 22:58, Sam James <sam@gentoo.org> wrote:
>> Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
>> amount of RAM available (uses amount declared as needed
>> in the ebuild). Typically should be ~2GB per job.
>> [snip]
> 
> I'm still not sure I grasp why we cannot make OOMs easier to discover
> from portage.
> 
> Most packages don't even use check-reqs, so your solution is very
> narrow (and I get why, because you get a lot of bug reports from the
> big packages that do use it.)
> 

Yeah, you've got the thinking here :)

> Can we write a build log analyzer?

I think this is a good idea, even if it's just a few hand-gathered regexes,
we can try improve the situation a bit, even for non-OOM (like
some common Perl-related failures).

> 
> -A
> 
> PS: If this was a global change I'd downvote it. It's only for
> check-reqs though and most packages don't use check-reqs; I don't
> really care. I'd be concerned about adopting this kind of approach
> wider; its very much a bandaid.
> 

Yep, I don't deny that, and I can't really say I'd support this
if it was global. The specific intent here is that check-reqs
consumers are generally big beasts and hence it's a direct,
targeted action to reduce bad bug reports and improve
the user experience, especially for newcomers who
end up hitting OOMs early on.

Best,
sam


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[David Seifert]

On Wed, 2022-01-05 at 09:28 +0100, Ulrich Mueller wrote:
> > > > > > On Tue, 04 Jan 2022, Sam James wrote:
> 
> > Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the
> > amount of RAM available (uses amount declared as needed
> > in the ebuild). Typically should be ~2GB per job.
> 
> Where does this number 2 GB come from? The amount of RAM strongly
> depends on the programming language and other factors, so I don't
> believe that there's one number that can be used for everything.
> (If only considering C and C++ 2 GB seems to be excessive, i.e. it
> will limit parallel build more than necessary. When linking, the
> number
> may be _much_ larger.)

https://wiki.gentoo.org/wiki/MAKEOPTS
   The rule of thumb is MAKEOPTS jobs should be the smaller number of: the
   size of RAM divided by 2GB or the number of threads the CPU has.

You're free to propose a better scheme, but try compiling some template-
heavy C++ code with 2GB/thread and -O3 and then WONTFIXing all the bug
reports. OTOH, if you emake -j1, people complain about stuff being
slower than glacial (https://bugs.gentoo.org/744088). Unless I see a
better solution, sam's solution is the best way to avoid these annoying,
rare, edge-case pitfalls.

> Also not sure if I understand the arithmetic. Shouldn't it use
> CHECKREQS_MEMORY as the basis of the calculation?
> 
> > +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS
> > +# @USER_VARIABLE
> > +# @DESCRIPTION:
> > +# Allow packages to reduce the number of multiprocessing (e.g.
> > make, ninja) jobs
> > +# to lower memory usage.
> > +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes}
> 
> If anything, the feature should be opt-in rather than opt-out.
> 
> Ulrich

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

On 2022.01.05 20:22, Sam James wrote:
> 
> 
> > On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>
> wrote:
> > 
> > Sam,
> > 
> > Do users with FEATURES=distcc still have to opt out of this
> > MAKEOPTS clamping?
> > 
> 
> Great point! I think we could add an exemption for that and make it a
> noop or warning-only.
> 
> Best,
> sam
> 
> 


Sam,

You are building a better mousetrap here. That's not a reason to try.

Do users of I_KNOW_WHAT_I_AM_DOING, who have already
opted to shoot themselves in both feet, get a free pass here?  

There are users who run emerge --jobs=X with MAKEOPTS='-jY"
and get firefox, thunderbird and libreoffice all building concurrently
as they allow X * Y MAKE threads, reduced by this proposed 
throttling, still triggering the OOM.

I don't think you can head that off beforehand. 

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
arm64

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[David Seifert]

On Wed, 2022-01-05 at 21:06 +0000, Roy Bamford wrote:
> On 2022.01.05 20:22, Sam James wrote:
> > 
> > 
> > > On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>
> > wrote:
> > > 
> > > Sam,
> > > 
> > > Do users with FEATURES=distcc still have to opt out of this
> > > MAKEOPTS clamping?
> > > 
> > 
> > Great point! I think we could add an exemption for that and make it
> > a
> > noop or warning-only.
> > 
> > Best,
> > sam
> > 
> > 
> 
> 
> Sam,
> 
> You are building a better mousetrap here. That's not a reason to try.
> 
> Do users of I_KNOW_WHAT_I_AM_DOING, who have already
> opted to shoot themselves in both feet, get a free pass here?  
> 
> There are users who run emerge --jobs=X with MAKEOPTS='-jY"
> and get firefox, thunderbird and libreoffice all building concurrently
> as they allow X * Y MAKE threads, reduced by this proposed 
> throttling, still triggering the OOM.
> 
> I don't think you can head that off beforehand. 
> 

What's your proposed alternative?

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1660 bytes --]

>>>>> On Wed, 05 Jan 2022, Sam James wrote:

>> On 5 Jan 2022, at 08:28, Ulrich Mueller <ulm@gentoo.org> wrote:
>> 
>> Where does this number 2 GB come from? The amount of RAM strongly
>> depends on the programming language and other factors, so I don't
>> believe that there's one number that can be used for everything.
>> (If only considering C and C++ 2 GB seems to be excessive, i.e. it
>> will limit parallel build more than necessary. When linking, the number
>> may be _much_ larger.)

> This is essentially "common law" (or "common lore" if you like!) and
> is well-accepted as a good rule of thumb.

Well, if the 2 GiB was a universal constant then it would be valid for
_all_ builds, not just the random subset inheriting check-reqs.eclass.

> The number being larger doesn't really make any difference here;
> the point is to help in cases where we're pretty sure there's going
> to be a problem (only users of check-reqs.eclass, and we can
> Introduce a variable to give ~RAM per job too if needed).

Yeah, having the ebuild specify an estimate for the average memory usage
per job would be more correct.

These are all band-aids however. What we would really need are options
like GNU parallel's --memfree and --memsuspend.

> This is also about reducing the number of support queries about
> builds which failed for trivial reasons, as someone who has to handle
> quite a lot of those (until such a time as we can implement better
> detection, but this is also a generally nice UX improvement -- as
> per what Florian/flow said).

But it would defeat the YAFIYGI principle that we commonly apply. :)

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 507 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Dale]

Roy Bamford wrote:
> On 2022.01.05 20:22, Sam James wrote:
>>
>>> On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>
>> wrote:
>>> Sam,
>>>
>>> Do users with FEATURES=distcc still have to opt out of this
>>> MAKEOPTS clamping?
>>>
>> Great point! I think we could add an exemption for that and make it a
>> noop or warning-only.
>>
>> Best,
>> sam
>>
>>
>
> Sam,
>
> You are building a better mousetrap here. That's not a reason to try.
>
> Do users of I_KNOW_WHAT_I_AM_DOING, who have already
> opted to shoot themselves in both feet, get a free pass here?  
>
> There are users who run emerge --jobs=X with MAKEOPTS='-jY"
> and get firefox, thunderbird and libreoffice all building concurrently
> as they allow X * Y MAKE threads, reduced by this proposed 
> throttling, still triggering the OOM.
>
> I don't think you can head that off beforehand. 
>


As a user, can I get a large +1 to that.  For me, it is usually
Seamonkey, Firefox, Libreoffice, that big qt package or some combination
of two or more of them.  I've had times where all of those just happen
to upgrade at the same time.  My solution was to put those on spinning
rust while using tmpfs for everything else and if needed, using
--exclude to put off one or more of them then do those later. 

While I like the general idea of this, and would love to see emerge be
able to handle it without failures, I'm not sure how emerge or it's
options can prevent it without slowing down other packages.  My
thinking, have emerge trigger when certain packages are in the upgrade
list and only allow one package in that list to update at a time.  For
example.  If Firefox, Libreoffice and that qt package are in the list of
upgrades, whichever hits first causes the others to wait until the first
one is done.  That will make it so only one large package is being
upgraded at a time and be able to have fast settings for other smaller
and less memory hungry packages as well. 

How to do that, I'm not a coder so no idea but I know there are some
awesome coders here who may can find a way.  If that idea is even
something that could be done. 

Since I rarely post here, keep up the great work.  :-D

Dale

:-)  :-) 

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On 2022.01.05 21:10, David Seifert wrote:
> On Wed, 2022-01-05 at 21:06 +0000, Roy Bamford wrote:
> > On 2022.01.05 20:22, Sam James wrote:
> > > 
> > > 
> > > > On 5 Jan 2022, at 19:02, Roy Bamford <neddyseagoon@gentoo.org>
> > > wrote:
> > > > 
> > > > Sam,
> > > > 
> > > > Do users with FEATURES=distcc still have to opt out of this
> > > > MAKEOPTS clamping?
> > > > 
> > > 
> > > Great point! I think we could add an exemption for that and make
> it
> > > a
> > > noop or warning-only.
> > > 
> > > Best,
> > > sam
> > > 
> > > 
> > 
> > 
> > Sam,
> > 
> > You are building a better mousetrap here. That's not a reason to
> try.
> > 
> > Do users of I_KNOW_WHAT_I_AM_DOING, who have already
> > opted to shoot themselves in both feet, get a free pass here?  
> > 
> > There are users who run emerge --jobs=X with MAKEOPTS='-jY"
> > and get firefox, thunderbird and libreoffice all building
> concurrently
> > as they allow X * Y MAKE threads, reduced by this proposed 
> > throttling, still triggering the OOM.
> > 
> > I don't think you can head that off beforehand. 
> > 
> 
> What's your proposed alternative?
> 
> 

I don't really have one ... hence the better mousetrap analogy at 
the start of my post.

grep -i killed

on the build log, if  portage can do that and tell the user the rules
of thumb to try to stop it in future. 

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
arm64

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage

[Kai Krakow]

Am Mi., 5. Jan. 2022 um 21:21 Uhr schrieb Sam James <sam@gentoo.org>:
>
>> On 5 Jan 2022, at 19:18, Kai Krakow <kai@kaishome.de> wrote:
>
>>> Am Mi., 5. Jan. 2022 um 19:22 Uhr schrieb Ulrich Mueller <ulm@gentoo.org>:
>
> [...]
>
>>> That applies to all parallel builds though, not only to ebuilds
>>> inheriting check-reqs.eclass. By tweaking MAKEOPTS, we're basically
>>> telling the user that the --jobs setting in their make.conf is wrong,
>>> in the first place.
>
>
>> Well, I'm using a safe combination of jobs and load-average, maybe the
>> documentation should be tweaked instead.
>
>
> I think "safe" is doing some heavy lifting here...

Well, works "safe" for me at least, but you're right.

>> I'm using
>> [...]
>
>
>> The "--jobs" parameter is mostly a safe-guard against "make" or
>> "emerge" overshooting the system resources which would happen if
>> running unconstrained without "--load-average". The latter parameter
>> OTOH tunes the parallel building processes automatically to the
>> available resources. If the system starves of memory, thus starts to
>> swap, load will increase, and make will reduce the jobs. It works
>> pretty well.
>
>> I've chosen the emerge loadavg limit slightly higher so a heavy ebuild
>> won't starve emerge from running configure phases of parallel ebuilds.
>
>
> ... because it's quite hard for this logic to work correctly enough
> of the time without jobserver integration (https://bugs.gentoo.org/692576).

Oh there's a bug report about this... I already wondered: Wouldn't it
be better if it had a global jobserver? OTOH, there are so many build
systems out there which parallelize building, and many of them won't
use a make jobserver but roll their own solution. So it looks a bit
futile on that side. That's why I've chosen the loadavg-based
approach.

> But indeed, I'd say you're not the target audience for this (but I appreciate
> the input).

Maybe not, I'm usually building in tmpfs (except huge source archives
with huge build artifacts), that means, I usually have plenty of RAM,
at least enough so it doesn't become the limiting factor.

But then again, what is the target audience? This proposal looks like
it tries to predict the future, and that's probably never going to
work right. Looking at the Github issue linked initially in the
thread, it looks like I /might/ be the target audience for packages
like qtwebkit because I'm building in tmpfs. The loadavg limiter does
quite well here unless a second huge ebuild becomes unpacked and built
in the tmpfs, at which point the system struggles to keep up and
starves from IO thrashing just to OOM portage a few moments later.
That's of course not due to the build jobs itself then, it's purely a
memory limitation. But for that reason I have configuration to build
such packages outside of tmpfs: While they usually work fine when
building just that package alone, it fails the very moment two of such
packages are built in parallel.

Maybe portage needs a job server that dynamically bumps the job
counter up or down based on current memory usage? Or "make" itself
could be patched to take that into account? But that's probably the
whole idea of the loadavg limiter. So I'd propose to at least mention
that in the documentation and examples, it seems to only be little
known.

Then again, if we run in a memory constrained system, it may be better
to parallelize ebuilds instead of build jobs to better make use of
combining light and heavy ebuild phases into the same time period.

Also, I'm not sure if 2 GB per job is the full picture - no matter if
that number is correct or isn't... Because usually the link phase of
packages like Chrome is the real RAM burner even with sane "jobs"
parameters. I've seen people failing to install these packages because
they didn't turn on swap, and then during the link phase, the compiler
took so much memory that it either froze the system for half an hour,
or OOMed. And at that stage, there's usually just this single compiler
process running (and maybe some small ones which almost use no memory
relative to that). And that doesn't get better with modern compilers
doing all sorts of global optimization stuff like LTO.

So maybe something like this could work (excluding the link phase):

If there's potentially running just one ebuild at a time (i.e. your
merge list has just one package), the effects of MAKEOPTS is quite
predictable. But if we potentially run more, we could carefully reduce
the number of jobs in MAKEOPTS before applying additional RAM
heuristics. And those heuristics probably should take the combination
of both emerge jobs and make jobs into account because potentially
that multiplies (unless 692576 is implemented).

Compiler and link flags may also be needed to take into account.

And maybe portage should take care of optionally serializing huge
packages and never build/unpack them at the same time. This would be a
huge winner for me so I would not have to manually configure things...
Something like PORTAGE_SERIALIZE_CONSTRAINED="1" to build at most one
package that has some RAM/storage warning vars in the ebuild. But
that's probably a different topic as it doesn't exactly target the
problem discussed here - and I'm also aware of this problem unlike the
target audience.


Regards,
Kai