[gentoo-dev] controlling src_test

[gentoo-dev] controlling src_test

[Ryan Hill]

There are several packages in portage (and even in base-system) that fail
in src_test when userpriv/usersandbox is enabled or disabled.  That is, some
testsuites fail when run as root and some fail if not run as root.

I'd like a simple consistent way to mark or handle these packages without
disabling tests altogether (RESTRICT=test).  As mentioned recently, checking
${FEATURES} in an ebuild is frowned upon, and it doesn't seem right to handle
this on a per-ebuild basis.  How would something like this best be implemented?
A split up RESTRICT (test_userpriv/test_nouserpriv)?  test.eclass? Something
else?  Looking at the bigger picture, are there any other situations where
finer-grained control over the test system would be helpful?

While I'm on the subject, I also thought it would be cool to have the option to
not die on a src_test failure, but instead to dump the log file and continue
on to the install phase.  I know this can be done per-ebuild, but it'd be
a useful global option.


-- 
                  fonts / wxWindows / gcc-porting / treecleaners
  9B81 6C9F E791 83BB 3AB3  5B2D E625 A073 8379 37E8 (0x837937E8)

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] controlling src_test

[Ravi Pinjala]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ryan Hill wrote:
> There are several packages in portage (and even in base-system) that fail
> in src_test when userpriv/usersandbox is enabled or disabled.  That is, some
> testsuites fail when run as root and some fail if not run as root.
> 
> I'd like a simple consistent way to mark or handle these packages without
> disabling tests altogether (RESTRICT=test).  As mentioned recently, checking
> ${FEATURES} in an ebuild is frowned upon, and it doesn't seem right to handle
> this on a per-ebuild basis.  How would something like this best be implemented?
> A split up RESTRICT (test_userpriv/test_nouserpriv)?  test.eclass? Something
> else?  Looking at the bigger picture, are there any other situations where
> finer-grained control over the test system would be helpful?
> 
> While I'm on the subject, I also thought it would be cool to have the option to
> not die on a src_test failure, but instead to dump the log file and continue
> on to the install phase.  I know this can be done per-ebuild, but it'd be
> a useful global option.
> 

I, for one, would like to be able to control whether or not to run tests
that take a huge amount of time to run. Some test suites are
ridiculously comprehensive, and if we could have an option to disable
only those, or even run a reduced test suite, that'd be pretty neat.

- --Ravi

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHBI2JsDgfoknqAqMRAj7+AKDY1wJYClm13wmxMWxgGxCfqzlfOwCgl2tT
qXwJ8Cz5HL+W1ois4IRzvOw=
=CjAo
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] controlling src_test

[Doug Goldstein]

Ravi Pinjala wrote:
> Ryan Hill wrote:
> > There are several packages in portage (and even in base-system) that
> fail
> > in src_test when userpriv/usersandbox is enabled or disabled.  That
> is, some
> > testsuites fail when run as root and some fail if not run as root.
>
> > I'd like a simple consistent way to mark or handle these packages
> without
> > disabling tests altogether (RESTRICT=test).  As mentioned recently,
> checking
> > ${FEATURES} in an ebuild is frowned upon, and it doesn't seem right
> to handle
> > this on a per-ebuild basis.  How would something like this best be
> implemented?
> > A split up RESTRICT (test_userpriv/test_nouserpriv)?  test.eclass?
> Something
> > else?  Looking at the bigger picture, are there any other situations
> where
> > finer-grained control over the test system would be helpful?
>
> > While I'm on the subject, I also thought it would be cool to have
> the option to
> > not die on a src_test failure, but instead to dump the log file and
> continue
> > on to the install phase.  I know this can be done per-ebuild, but
> it'd be
> > a useful global option.
>
>
> I, for one, would like to be able to control whether or not to run tests
> that take a huge amount of time to run. Some test suites are
> ridiculously comprehensive, and if we could have an option to disable
> only those, or even run a reduced test suite, that'd be pretty neat.
>
> --Ravi
>
Who and what determines if a test is overly comprehensive and takes too
long to run?
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] controlling src_test

[Thomas Anderson]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]

On Thursday 04 October 2007 09:36:29 Doug Goldstein wrote:
> Ravi Pinjala wrote:
> > Ryan Hill wrote:
> > > There are several packages in portage (and even in base-system) that
> >
> > fail
> >
> > > in src_test when userpriv/usersandbox is enabled or disabled.  That
> >
> > is, some
> >
> > > testsuites fail when run as root and some fail if not run as root.
> > >
> > > I'd like a simple consistent way to mark or handle these packages
> >
> > without
> >
> > > disabling tests altogether (RESTRICT=test).  As mentioned recently,
> >
> > checking
> >
> > > ${FEATURES} in an ebuild is frowned upon, and it doesn't seem right
> >
> > to handle
> >
> > > this on a per-ebuild basis.  How would something like this best be
> >
> > implemented?
> >
> > > A split up RESTRICT (test_userpriv/test_nouserpriv)?  test.eclass?
> >
> > Something
> >
> > > else?  Looking at the bigger picture, are there any other situations
> >
> > where
> >
> > > finer-grained control over the test system would be helpful?
> > >
> > > While I'm on the subject, I also thought it would be cool to have
> >
> > the option to
> >
> > > not die on a src_test failure, but instead to dump the log file and
> >
> > continue
> >
> > > on to the install phase.  I know this can be done per-ebuild, but
> >
> > it'd be
> >
> > > a useful global option.
> >
> > I, for one, would like to be able to control whether or not to run tests
> > that take a huge amount of time to run. Some test suites are
> > ridiculously comprehensive, and if we could have an option to disable
> > only those, or even run a reduced test suite, that'd be pretty neat.
> >
> > --Ravi
>
> Who and what determines if a test is overly comprehensive and takes too
> long to run?

I think most everybody agrees that boost's tests are overly comprehensive. As 
for others like mysql,  a long test may be necessary to ensure everything is 
working properly.

-- 
2.6.22-gentoo-r8

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] controlling src_test

[Rémi Cardona]

Ravi Pinjala a écrit :
> I, for one, would like to be able to control whether or not to run tests
> that take a huge amount of time to run. Some test suites are
> ridiculously comprehensive, and if we could have an option to disable
> only those, or even run a reduced test suite, that'd be pretty neat.

We've already had this discussion before [1] and there is no 
one-size-fits-all solution for tagging tests.

Ryan's plan only proposes to enable/disable tests based on whether they 
can run inside a sandbox. That much is doable. Trying to quantify the 
time a test suite will take to complete is impossible : different 
arches, vastly different CPU clock speeds, same for HDDs sizes and 
speeds, ... is impossible.

Basically, it boils down to "if you're running FEATURES=test, you should 
know what you're doing".

+1 on Ryan's idea

Rémi

[1] http://archives.gentoo.org/gentoo-dev/msg_145157.xml
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: controlling src_test

[Ryan Hill]

Rémi Cardona wrote:
> Ravi Pinjala a écrit :
>> I, for one, would like to be able to control whether or not to run tests
>> that take a huge amount of time to run. Some test suites are
>> ridiculously comprehensive, and if we could have an option to disable
>> only those, or even run a reduced test suite, that'd be pretty neat.
> 
> We've already had this discussion before [1] and there is no
> one-size-fits-all solution for tagging tests.
> 
> Ryan's plan only proposes to enable/disable tests based on whether they
> can run inside a sandbox. That much is doable. Trying to quantify the
> time a test suite will take to complete is impossible : different
> arches, vastly different CPU clock speeds, same for HDDs sizes and
> speeds, ... is impossible.

I usually disable tests on a per-package basis using /etc/portage/env
entries.

dirtyepic@tycho ~ $ cat /etc/portage/env/sci-libs/fftw
NEWFEATURES=

for f in ${FEATURES}; do
    if [[ ! $f == "test" ]]; then
        NEWFEATURES="${NEWFEATURES} $f"
    fi
done

FEATURES="${NEWFEATURES}"


-- 
                  fonts / wxWindows / gcc-porting / treecleaners
  EFFD 380E 047A 4B51 D2BD  C64F 8AA8 8346 F9A4 0662 (0xF9A40662)

-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: controlling src_test

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

Ryan Hill wrote:
> Rémi Cardona wrote:
>> Ravi Pinjala a écrit :
>>> I, for one, would like to be able to control whether or not to run tests
>>> that take a huge amount of time to run. Some test suites are
>>> ridiculously comprehensive, and if we could have an option to disable
>>> only those, or even run a reduced test suite, that'd be pretty neat.
>> We've already had this discussion before [1] and there is no
>> one-size-fits-all solution for tagging tests.
>>
>> Ryan's plan only proposes to enable/disable tests based on whether they
>> can run inside a sandbox. That much is doable. Trying to quantify the
>> time a test suite will take to complete is impossible : different
>> arches, vastly different CPU clock speeds, same for HDDs sizes and
>> speeds, ... is impossible.
> 
> I usually disable tests on a per-package basis using /etc/portage/env
> entries.
> 
> dirtyepic@tycho ~ $ cat /etc/portage/env/sci-libs/fftw
> NEWFEATURES=
> 
> for f in ${FEATURES}; do
>     if [[ ! $f == "test" ]]; then
>         NEWFEATURES="${NEWFEATURES} $f"
>     fi
> done
> 
> FEATURES="${NEWFEATURES}"

here's a crappy little script to automate it.

-- 
                  fonts / wxWindows / gcc-porting / treecleaners
  EFFD 380E 047A 4B51 D2BD  C64F 8AA8 8346 F9A4 0662 (0xF9A40662)

[-- Attachment #2: defeature --]
[-- Type: text/plain, Size: 1457 bytes --]

#!/bin/bash -
#
# defeature - disable a FEATURE on a per-package basis.
#
# <dirtyepic@gentoo.org>

source /usr/lib/portage/bin/isolated-functions.sh

defeature() {
	if [[ ! $# -eq 2 ]]; then
		echo
		eerror "Usage:  $(basename $0) <FEATURE> <pkgname>"
		echo
		exit 1
	fi

	local pkgcat pkgname

	pkgcat=${!#%%/*}
	pkgname=${!###/*}

	if [[ $(equery w =${pkgname} 2>&1 | grep "No masked" ) ]]; then
		echo
		eerror "No package found matching \"${pkgname}\"."
		echo
		exit 1
	fi

	[[ $pkgcat == $pkgname ]] && pkgcat=

	if [[ -z ${pkgcat} ]]; then

		# Need a better way to get category - use udept for now

		# check for ambiguous package names
		if [[ $(dep -1c ${pkgname} | awk 'END { print NR }') -gt 1 ]]; then
			echo
			eerror "Found multiple categories for \"${pkgname}\"."
			eerror "Please use a fully qualified package name (cat/package)."
			echo
			exit 1
		fi

		pkgcat="$(dep -1c ${pkgname})"
	    # strip whitespace
		pkgcat="${pkgcat// /}"
	fi

	[[ ! -d /etc/portage/env/${pkgcat} ]] \
		&& sudo mkdir -p /etc/portage/env/${pkgcat}

	# note that this overwrites anything already in the file
	cat > /etc/portage/env/${pkgcat}/${pkgname} <<-EOF
		NEWFEATURES=

		for f in \${FEATURES}; do
		    if [[ ! \$f == "${1}" ]]; then
				NEWFEATURES="\${NEWFEATURES} \$f"
			fi
		done

		FEATURES="\${NEWFEATURES}"
	EOF

	if [[ $? -eq 0 ]]; then
		echo
		einfo "${1} FEATURE disabled for ${pkgcat}/${pkgname}."
		echo
	fi

	exit 0
}

defeature "$@"

Re: [gentoo-dev] Re: controlling src_test

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

On Thu, 2007-10-04 at 13:47 -0600, Ryan Hill wrote:
> > I usually disable tests on a per-package basis using /etc/portage/env
> > entries.
> > 
> > dirtyepic@tycho ~ $ cat /etc/portage/env/sci-libs/fftw
> > NEWFEATURES=
> > 
> > for f in ${FEATURES}; do
> >     if [[ ! $f == "test" ]]; then
> >         NEWFEATURES="${NEWFEATURES} $f"
> >     fi
> > done
> > 
> > FEATURES="${NEWFEATURES}"
> 
> here's a crappy little script to automate it.

See, I think this sort of thing is what we should be adding into
gentoolkit.  It is something that is very useful, but not required for
running a Gentoo system.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Foundation Trustee
Gentoo Foundation

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: controlling src_test

[Rémi Cardona]

Ryan Hill wrote:
> here's a crappy little script to automate it.

This is really cool. /me had somehow missed the lesson on /etc/portage/env

Cheers :)

Rémi
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: controlling src_test

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 705 bytes --]

Ryan Hill wrote:
> 
> here's a crappy little script to automate it.
> 

here's take two.  if someone could tell me a simple way to get the
category given a package name, including a way to detect ambiguous
names, i could drop the dependency on app-portage/udept.

anyways it now supports disabling more than one FEATURE and defaults
to appending to the env file with an -o option for overwriting.

it won't rape your dog though it may try.  it could also probably be
rewritten using nothing but pure sh, awk, perl, or GWBASIC, but i
think it does what i want it to. ;)

-- 
                  fonts / wxWindows / gcc-porting / treecleaners
  EFFD 380E 047A 4B51 D2BD  C64F 8AA8 8346 F9A4 0662 (0xF9A40662)

[-- Attachment #2: defeature --]
[-- Type: text/plain, Size: 2110 bytes --]

#!/bin/bash -
#
# defeature - disable a FEATURE on a per-package basis.
#
# <dirtyepic@gentoo.org>

source /usr/lib/portage/bin/isolated-functions.sh

defeature() {
	if [[ ! -w /etc/portage ]]; then
		echo
		eerror "You don't have write permissions for /etc/portage."
		echo
		exit 1
	fi

	local opt overwrite

	while getopts o opt; do
		case ${opt} in
			o)		overwrite="true" ;;
			'?')	print_usage ;;
		esac
	done

	shift $((OPTIND - 1))

	[[ $# -lt 2 ]] && print_usage

	local pkgcat pkgname

	pkgcat=${!#%%/*}
	pkgname=${!###/*}

	if [[ $(equery w =${pkgname} 2>&1 | grep "No masked" ) ]]; then
		echo
		eerror "No package found matching \"${pkgname}\"."
		echo
		exit 1
	fi

	[[ $pkgcat == $pkgname ]] && pkgcat=

	if [[ -z ${pkgcat} ]]; then

		# Need a better way to get category - use app-portage/udept for now

		# check for ambiguous package names
		if [[ $(dep -1c ${pkgname} | awk 'END { print NR }') -gt 1 ]]; then
			echo
			eerror "Found multiple categories for \"${pkgname}\"."
			eerror "Please use a fully qualified package name (cat/package)."
			echo
			exit 1
		fi

		pkgcat="$(dep -1c ${pkgname})"
	    # strip whitespace
		pkgcat="${pkgcat// /}"
	fi

	[[ ! -d /etc/portage/env/${pkgcat} ]] && mkdir -p /etc/portage/env/${pkgcat}

	local feathead featloop featlist

	featheader="FEATURES=\" \${FEATURES}\""

	while [[ ${#} -gt 1 ]]; do
		featlist="${featlist} ${1}"
		featloop="${featloop}\nFEATURES=\${FEATURES/ ${1}/}"
		shift
	done

	if [[ ${overwrite} ]]; then
		echo -e "${featheader}${featloop}" > /etc/portage/env/${pkgcat}/${pkgname}
	else
        echo -e "${featheader}${featloop}" >> /etc/portage/env/${pkgcat}/${pkgname}
	fi

	if [[ $? -eq 0 ]]; then
		echo
		einfo "Disabled${featlist} FEATURE(s) for ${pkgcat}/${pkgname}."
		einfo
		einfo "To re-enable default behaviour for this package, delete"
		einfo "/etc/portage/env/${pkgcat}/${pkgname}."
		echo
	fi

	exit 0
}

print_usage() {
		echo
		eerror "Usage:  $(basename $0) [-o] <FEATURE> [FEATURE] <pkgname>"
		eerror
		eerror "        -o    overwrite current settings rather than append"
		echo
		exit 1
}

defeature "$@"

Re: [gentoo-dev] Re: controlling src_test

[Alin Năstac]

[-- Attachment #1: Type: text/plain, Size: 377 bytes --]

Ryan Hill wrote:
> NEWFEATURES=
>
> for f in ${FEATURES}; do
>     if [[ ! $f == "test" ]]; then
>         NEWFEATURES="${NEWFEATURES} $f"
>     fi
> done
>
> FEATURES="${NEWFEATURES}"
>
>   
There is a simpler way to remove a word from an environment variable:
FEATURES=" ${FEATURES}" # make sure every word is prefixed by a space
FEATURES=${FEATURES/ test/}


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

[gentoo-dev] Re: controlling src_test

[Ryan Hill]

Alin Năstac wrote:
> Ryan Hill wrote:
>> NEWFEATURES=
>>
>> for f in ${FEATURES}; do
>>     if [[ ! $f == "test" ]]; then
>>         NEWFEATURES="${NEWFEATURES} $f"
>>     fi
>> done
>>
>> FEATURES="${NEWFEATURES}"
>>
>>   
> There is a simpler way to remove a word from an environment variable:
> FEATURES=" ${FEATURES}" # make sure every word is prefixed by a space
> FEATURES=${FEATURES/ test/}

Doh.  I tried ${FEATURES## test} but didn't think of substitution.

This actually solves another problem for me.  Thanks.

-- 
                  fonts / wxWindows / gcc-porting / treecleaners
  EFFD 380E 047A 4B51 D2BD  C64F 8AA8 8346 F9A4 0662 (0xF9A40662)

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] controlling src_test

[Marius Mauch]

On Wed, 03 Oct 2007 19:50:01 -0600
Ryan Hill <dirtyepic@gentoo.org> wrote:

> There are several packages in portage (and even in base-system) that
> fail in src_test when userpriv/usersandbox is enabled or disabled.
> That is, some testsuites fail when run as root and some fail if not
> run as root.
> 
> I'd like a simple consistent way to mark or handle these packages
> without disabling tests altogether (RESTRICT=test).  As mentioned
> recently, checking ${FEATURES} in an ebuild is frowned upon, and it
> doesn't seem right to handle this on a per-ebuild basis.  How would
> something like this best be implemented? A split up RESTRICT
> (test_userpriv/test_nouserpriv)?  test.eclass? Something else?
> Looking at the bigger picture, are there any other situations where
> finer-grained control over the test system would be helpful?

See http://bugs.gentoo.org/show_bug.cgi?id=159876 for some suggestions
for the "test-only-as-root" case. IMO ebuilds should simply test for
the actual capabilities they need in src_test (like uid) instead of
more abstract things like userpriv. If such tests can be used in
several ebuilds an eclass can help to standardize them, but I don't see
a reason to move that logic into the package manager unless those cases
are extremely common.
As for fine-grained user-control, it's a question of quantification as
discussed previously, which isn't easy to solve, or you have to
en-/disable things manually and the issue is part tf the
per-package-env-variables problem (btw, the /etc/portage/env trick only
works because the default src_test in ebuild.sh has the
otherwise redundant FEATURES check which was discussed a few days ago in
one of the commit reviews)

Marius
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] controlling src_test

[Bo Ørsted Andresen]

[-- Attachment #1: Type: text/plain, Size: 420 bytes --]

On Friday 05 October 2007 01:14:32 Marius Mauch wrote:
> (btw, the /etc/portage/env trick only works because the default src_test in
> ebuild.sh has the otherwise redundant FEATURES check which was discussed a
> few days ago in one of the commit reviews)

src_test() is not called in dyn_test() unless test is in FEATURES. Therefore 
the FEATURES check in the default src_test() is redundant.

-- 
Bo Andresen

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]