[gentoo-user] Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[gentoo-user] Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Lie Ryan]

I'm running with full system FEATURES="test" on, and I have a couple of
programs that depended on dev-libs/boost. The boost testsuite always
fails in my computer due to insufficient disk space, I usually simply
skip the test for boost and just go on with the merge. But today, I
decided to let the testsuite run to completion; so in preparation for
that, I plugged in an external harddisk and made it so that
/var/tmp/portage points to an empty disk image in the external harddrive.

This setup works ok, and the testsuite is still running, however I saw
now that the disk image's is now taking ~18 GB (and counting) while "du
-sh" on /var/tmp/portage counted ~13GB.

So, the question is, has anyone successfully compiled and run
FEATURES="test" on boost and knows how much space the tests eat up in
the end?

I am suspecting of the possibility that maybe a testsuite gets into an
infinite loop while writing a file or something constantly eats up
diskspace. Or is it just that boost has an outrageously too extensive
testsuite and it will turn out ok if I just left it to run.

Re: [gentoo-user] Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Paul Hartman]

On Sun, Apr 4, 2010 at 5:00 PM, Lie Ryan <lie.1296@gmail.com> wrote:
> I'm running with full system FEATURES="test" on, and I have a couple of
> programs that depended on dev-libs/boost. The boost testsuite always
> fails in my computer due to insufficient disk space, I usually simply
> skip the test for boost and just go on with the merge. But today, I
> decided to let the testsuite run to completion; so in preparation for
> that, I plugged in an external harddisk and made it so that
> /var/tmp/portage points to an empty disk image in the external harddrive.
>
> This setup works ok, and the testsuite is still running, however I saw
> now that the disk image's is now taking ~18 GB (and counting) while "du
> -sh" on /var/tmp/portage counted ~13GB.
>
> So, the question is, has anyone successfully compiled and run
> FEATURES="test" on boost and knows how much space the tests eat up in
> the end?
>
> I am suspecting of the possibility that maybe a testsuite gets into an
> infinite loop while writing a file or something constantly eats up
> diskspace. Or is it just that boost has an outrageously too extensive
> testsuite and it will turn out ok if I just left it to run.

I'm trying it now, I have 64gb free in /var/tmp so I hope that's enough... :)

Re: [gentoo-user] Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Paul Hartman]

On Mon, Apr 5, 2010 at 9:15 AM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Sun, Apr 4, 2010 at 5:00 PM, Lie Ryan <lie.1296@gmail.com> wrote:
>> I'm running with full system FEATURES="test" on, and I have a couple of
>> programs that depended on dev-libs/boost. The boost testsuite always
>> fails in my computer due to insufficient disk space, I usually simply
>> skip the test for boost and just go on with the merge. But today, I
>> decided to let the testsuite run to completion; so in preparation for
>> that, I plugged in an external harddisk and made it so that
>> /var/tmp/portage points to an empty disk image in the external harddrive.
>>
>> This setup works ok, and the testsuite is still running, however I saw
>> now that the disk image's is now taking ~18 GB (and counting) while "du
>> -sh" on /var/tmp/portage counted ~13GB.
>>
>> So, the question is, has anyone successfully compiled and run
>> FEATURES="test" on boost and knows how much space the tests eat up in
>> the end?
>>
>> I am suspecting of the possibility that maybe a testsuite gets into an
>> infinite loop while writing a file or something constantly eats up
>> diskspace. Or is it just that boost has an outrageously too extensive
>> testsuite and it will turn out ok if I just left it to run.
>
> I'm trying it now, I have 64gb free in /var/tmp so I hope that's enough... :)
>

After almost 1.5 hours it is at its 22000th target and using 14G of
/var/tmp so far... I'll keep waiting.

Re: [gentoo-user] Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Paul Hartman]

On Mon, Apr 5, 2010 at 10:53 AM, Paul Hartman
<paul.hartman+gentoo@gmail.com> wrote:
> On Mon, Apr 5, 2010 at 9:15 AM, Paul Hartman
> <paul.hartman+gentoo@gmail.com> wrote:
>> On Sun, Apr 4, 2010 at 5:00 PM, Lie Ryan <lie.1296@gmail.com> wrote:
>>> I'm running with full system FEATURES="test" on, and I have a couple of
>>> programs that depended on dev-libs/boost. The boost testsuite always
>>> fails in my computer due to insufficient disk space, I usually simply
>>> skip the test for boost and just go on with the merge. But today, I
>>> decided to let the testsuite run to completion; so in preparation for
>>> that, I plugged in an external harddisk and made it so that
>>> /var/tmp/portage points to an empty disk image in the external harddrive.
>>>
>>> This setup works ok, and the testsuite is still running, however I saw
>>> now that the disk image's is now taking ~18 GB (and counting) while "du
>>> -sh" on /var/tmp/portage counted ~13GB.
>>>
>>> So, the question is, has anyone successfully compiled and run
>>> FEATURES="test" on boost and knows how much space the tests eat up in
>>> the end?
>>>
>>> I am suspecting of the possibility that maybe a testsuite gets into an
>>> infinite loop while writing a file or something constantly eats up
>>> diskspace. Or is it just that boost has an outrageously too extensive
>>> testsuite and it will turn out ok if I just left it to run.
>>
>> I'm trying it now, I have 64gb free in /var/tmp so I hope that's enough... :)
>>
>
> After almost 1.5 hours it is at its 22000th target and using 14G of
> /var/tmp so far... I'll keep waiting.
>

It finished, successfully. 1 hour 52 minutes (normal compile of boost
without testing takes about 3 minutes). Peak disk usage was about 20G
when i spot-checked...

The ebuild checks for 1024M free, maybe they need to change that to
check for 20G if testing?

[gentoo-user] Re: Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Lie Ryan]

On 04/06/10 02:43, Paul Hartman wrote:
> On Mon, Apr 5, 2010 at 10:53 AM, Paul Hartman
> <paul.hartman+gentoo@gmail.com> wrote:
>> On Mon, Apr 5, 2010 at 9:15 AM, Paul Hartman
>> <paul.hartman+gentoo@gmail.com> wrote:
>>> On Sun, Apr 4, 2010 at 5:00 PM, Lie Ryan <lie.1296@gmail.com> wrote:
>>>> I'm running with full system FEATURES="test" on, and I have a couple of
>>>> programs that depended on dev-libs/boost. The boost testsuite always
>>>> fails in my computer due to insufficient disk space, I usually simply
>>>> skip the test for boost and just go on with the merge. But today, I
>>>> decided to let the testsuite run to completion; so in preparation for
>>>> that, I plugged in an external harddisk and made it so that
>>>> /var/tmp/portage points to an empty disk image in the external harddrive.
>>>>
>>>> This setup works ok, and the testsuite is still running, however I saw
>>>> now that the disk image's is now taking ~18 GB (and counting) while "du
>>>> -sh" on /var/tmp/portage counted ~13GB.
>>>>
>>>> So, the question is, has anyone successfully compiled and run
>>>> FEATURES="test" on boost and knows how much space the tests eat up in
>>>> the end?
>>>>
>>>> I am suspecting of the possibility that maybe a testsuite gets into an
>>>> infinite loop while writing a file or something constantly eats up
>>>> diskspace. Or is it just that boost has an outrageously too extensive
>>>> testsuite and it will turn out ok if I just left it to run.
>>>
>>> I'm trying it now, I have 64gb free in /var/tmp so I hope that's enough... :)
>>>
>>
>> After almost 1.5 hours it is at its 22000th target and using 14G of
>> /var/tmp so far... I'll keep waiting.
>>
> 
> It finished, successfully. 1 hour 52 minutes (normal compile of boost
> without testing takes about 3 minutes). Peak disk usage was about 20G
> when i spot-checked...

Finished too (sorry didn't post the update earlier). And passed the test
too, great. Thank you for trying it as well!

Final count: the disk image's size is now ~22G but I don't know how
large the real disk usage in the end is since portage cleaned the
/var/tmp/portage. That's one hell of a test!

> The ebuild checks for 1024M free, maybe they need to change that to
> check for 20G if testing?

I think so too. Let's see if I can file a bug on that. Done:
http://bugs.gentoo.org/show_bug.cgi?id=313315

---

Anyway, I've been thinking about this for some time that turning on
FEATURES="test" globally seems quite impractical for many users due to
some test suites that:

- takes a disproportionally huge amount of time to finish
- takes a huge amount of harddisk
- takes a huge amount of memory
- pulled out optional dependencies used only for testing
- ships with a test that unconditionally fails (e.g. unimplemented
feature) but not marking it "expected failure"
- other behavioral problems (using network, restarting, etc though I've
never seen these sort of crime yet as of now, probably they're already
filtered before it reaches me)

Due to this problem, I think portage could have a test policy feature so
people can have finer control to filter out test suites that they don't
want to run. This way globally FEATURES="test" can be more feasible for
most users (and probably can sometime be turned on by default).

So is there anyone here that actually wanted to run FEATURES="test"
globally, but are turned off when experiencing the problems it brings?

Do you think if we want to hack this policy feature into portage or
emerge, what's the best option? Using USE-flags won't require much
change to emerge and portage but is quite inflexible; or new variables
in ebuild file; or a separate (optional) test description file that
portage will read and compare with the system's policy. Have better
alternative?

Re: [gentoo-user] Re: Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

On Tue, 06 Apr 2010 10:11:02 +1000, Lie Ryan wrote:

> Anyway, I've been thinking about this for some time that turning on
> FEATURES="test" globally seems quite impractical for many users

FEATURES=test is not meant to be used by users, it is a developer
setting, and they would only enable it for packages they maintain and
then only when they ant to run the tests.

> Due to this problem, I think portage could have a test policy feature so
> people can have finer control to filter out test suites that they don't
> want to run. This way globally FEATURES="test" can be more feasible for
> most users (and probably can sometime be turned on by default).

You can set features on a per-package basis by putting FEATURES="blah"
into /etc/portage/env/category/package.


-- 
Neil Bothwick

Pepperami. Its a bit of an animal.
What animal & what bit?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Harry Putnam]

Neil Bothwick <neil@digimed.co.uk> writes:

[...]

> You can set features on a per-package basis by putting FEATURES="blah"
> into /etc/portage/env/category/package.

If that would also work for something like always using a specific
EXTRA_ECONF for a certain package:

  EXTRA_ECONF="--enable-rootcommit"  <for the cvs package>

Can you show an example of the necessary syntax? 

Re: [gentoo-user] Re: Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 953 bytes --]

On Tue, 06 Apr 2010 11:17:14 -0500, Harry Putnam wrote:

> > You can set features on a per-package basis by putting FEATURES="blah"
> > into /etc/portage/env/category/package.  
> 
> If that would also work for something like always using a specific
> EXTRA_ECONF for a certain package:
> 
>   EXTRA_ECONF="--enable-rootcommit"  <for the cvs package>

It would.

> Can you show an example of the necessary syntax? 

Just put the variable assignment in the file, it is sourced by bash when
the ebuild is parsed, so most things that can go in an ebuild can go
here. Usually it is used to override settings or set EXTRA_ECONF but you
can use it to redefine the ebuild functions. Some people put a custom
src_unpack() in here when they want to apply a patch, rather than
putting a modified ebuild in an overlay. 


-- 
Neil Bothwick

Picard: 'What do the sensors say Mr Data?'
Data:   'They tell us that we can't say "F*ck" Sir."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Harry Putnam]

Neil Bothwick <neil@digimed.co.uk> writes:

> On Tue, 06 Apr 2010 11:17:14 -0500, Harry Putnam wrote:
>
>> > You can set features on a per-package basis by putting FEATURES="blah"
>> > into /etc/portage/env/category/package.  
>> 
>> If that would also work for something like always using a specific
>> EXTRA_ECONF for a certain package:
>> 
>>   EXTRA_ECONF="--enable-rootcommit"  <for the cvs package>
>
> It would.
>
>> Can you show an example of the necessary syntax? 
>
> Just put the variable assignment in the file, it is sourced by bash when
> the ebuild is parsed, so most things that can go in an ebuild can go
> here. Usually it is used to override settings or set EXTRA_ECONF but you
> can use it to redefine the ebuild functions. Some people put a custom
> src_unpack() in here when they want to apply a patch, rather than
> putting a modified ebuild in an overlay. 

Ahh very helpful, thank you.  Especially about putting custom
src_unpack stuff.  Fussing with creating a new ebuild is a pain to us
non devel types.

But one thing is unclear.  You say: 
`Just put the variable assignment in the file'

You don't mean without reference to a specific package do you.
Like:
  cat /etc/portage/env/category/package
  EXTRA_ECONF="--enable-rootcommit"

So is it:
  cat /etc/portage/env/category/package
  dev-util/cvs  EXTRA_ECONF="--enable-rootcommit"

Or something else?

Re: [gentoo-user] Re: Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]

On Tue, 06 Apr 2010 15:17:27 -0500, Harry Putnam wrote:

>  Just put the variable assignment in the file, it is sourced by bash
> when
> > the ebuild is parsed, so most things that can go in an ebuild can go
> > here. Usually it is used to override settings or set EXTRA_ECONF but
> > you can use it to redefine the ebuild functions. Some people put a
> > custom src_unpack() in here when they want to apply a patch, rather
> > than putting a modified ebuild in an overlay.   
> 
> Ahh very helpful, thank you.  Especially about putting custom
> src_unpack stuff.  Fussing with creating a new ebuild is a pain to us
> non devel types.

I find copying the existing ebuild to an overlay and adding one epatch
line a lot easier than writing a custom src_unpack() function.

> But one thing is unclear.  You say: 
> `Just put the variable assignment in the file'
> 
> You don't mean without reference to a specific package do you.
> Like:
>   cat /etc/portage/env/category/package
>   EXTRA_ECONF="--enable-rootcommit"

Yes I do.

> So is it:
>  cat /etc/portage/env/category/package
>  dev-util/cvs  EXTRA_ECONF="--enable-rootcommit"

No, it is

$ cat /etc/portage/env/dev-util/cvs
EXTRA_ECONF="--enable-rootcommit"


-- 
Neil Bothwick

"We demand rigidly defined areas of doubt and uncertainty!"

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-user] Re: Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Lie Ryan]

On 04/06/10 17:23, Neil Bothwick wrote:
> On Tue, 06 Apr 2010 10:11:02 +1000, Lie Ryan wrote:
> 
>> Anyway, I've been thinking about this for some time that turning on
>> FEATURES="test" globally seems quite impractical for many users
> 
> FEATURES=test is not meant to be used by users, it is a developer
> setting, and they would only enable it for packages they maintain and
> then only when they ant to run the tests.

But most developers do not have the resources to test on all
combinations of platforms. If the barrier for FEATURES="test" can be
lowered, then everyone that wants to be a global tester can do it
without sacrificing too muchs (plus they can control how much time they
want to contribute) and this benefits all open-source software as a
whole. Lowering barrier for testing also encourages developers to write
unittest who would otherwise hand-waving it since they now know their
unittest will really be testing the program's true correctness instead
of an platform dependent correctness. Probably enabling "test" by
default is too much to ask though.

>> Due to this problem, I think portage could have a test policy feature so
>> people can have finer control to filter out test suites that they don't
>> want to run. This way globally FEATURES="test" can be more feasible for
>> most users (and probably can sometime be turned on by default).
> 
> You can set features on a per-package basis by putting FEATURES="blah"
> into /etc/portage/env/category/package.

Re: [gentoo-user] Re: Anyone ever emerged dev-libs/boost with FEATURES="test" and finished?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

On Wed, 07 Apr 2010 05:51:03 +1000, Lie Ryan wrote:

> > FEATURES=test is not meant to be used by users, it is a developer
> > setting, and they would only enable it for packages they maintain and
> > then only when they ant to run the tests.  
> 
> But most developers do not have the resources to test on all
> combinations of platforms.

That's why Gentoo has arch-testers.


-- 
Neil Bothwick

If you give a man a fish, he's fed for a day.
If you teach a man to fish, he'll buy a silly hat.
If you talk about fish to a starving man, you're a consultant.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]