[gentoo-dev] RFC: Block ebuilds installing tests to ${D} by default

[gentoo-dev] RFC: Block ebuilds installing tests to ${D} by default

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1176 bytes --]

TL;DR:
I'd like to propose a change where packages should NOT install their
tests to ${D} by default. Such an install may optionally enabled with
USE=test, which should be decoupled from FEATURES=test. Or depending on
the color of the bikeshed, we add something new like USE=install-tests.

Background:
Python packages install a number of _test.py files, and related .pyc
files. The files are generally useful for running tests after the
package is installed, and may have additional testing dependencies that
are not installed via RDEPEND.

As an example, on the livegui install media, these files take 100MB+
before squashfs compression.

Some users MAY wish to verify that a package continues to function
correctly, and they should have the USE=test dependencies available at
runtime, and the tests installed.

Such post-install testing may also require other files to be present, to
configure the test suite runs.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation President & Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] RFC: Block ebuilds installing tests to ${D} by default

[Mike Gilbert]

On Thu, Feb 1, 2024 at 3:15 AM Robin H. Johnson <robbat2@gentoo.org> wrote:
>
> TL;DR:
> I'd like to propose a change where packages should NOT install their
> tests to ${D} by default. Such an install may optionally enabled with
> USE=test, which should be decoupled from FEATURES=test. Or depending on
> the color of the bikeshed, we add something new like USE=install-tests.
>
> Background:
> Python packages install a number of _test.py files, and related .pyc
> files. The files are generally useful for running tests after the
> package is installed, and may have additional testing dependencies that
> are not installed via RDEPEND.

It sounds like you want to resurrect bug 531648.

https://bugs.gentoo.org/531648

Re: [gentoo-dev] RFC: Block ebuilds installing tests to ${D} by default

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3865 bytes --]

On Thu, 2024-02-01 at 08:15 +0000, Robin H. Johnson wrote:
> TL;DR:
> I'd like to propose a change where packages should NOT install their
> tests to ${D} by default.

That sounds like reverting a lot of effort that has been cleaned up all
over the past years.

>  Such an install may optionally enabled with
> USE=test, which should be decoupled from FEATURES=test.

This would prevent us from detecting files accidentally left over by
FEATURES=test, as ago does right now.  A few times this was actually
helpful in identifying packages whose test suites wrote temporary files
into site-packages directory, and they effectively ended up being
installed.

>  Or depending on
> the color of the bikeshed, we add something new like USE=install-tests.
> 
> Background:
> Python packages install a number of _test.py files, and related .pyc
> files. The files are generally useful for running tests after the
> package is installed, and may have additional testing dependencies that
> are not installed via RDEPEND.

This is not really relevant.  This is the case of optional runtime
dependencies (i.e. you don't have to have them installed when installing
the test suite in question), and per policy PG-0001 these can be
expressed using postinst messages or likewise.

> As an example, on the livegui install media, these files take 100MB+
> before squashfs compression.
> 
> Some users MAY wish to verify that a package continues to function
> correctly, and they should have the USE=test dependencies available at
> runtime, and the tests installed.
> 
> Such post-install testing may also require other files to be present, to
> configure the test suite runs.

I suppose you are referring to dev-lang/python here.  Unfortunately,
removing tests from it is a non-trivial problem.  As I've mentioned to
you before, there are packages that actually import modules form the
test directory.

Remember that Gentoo has a policy of following upstream, and this policy
is specifically targeted towards developers who expect Gentoo to
be a good baseline environment for developing packages.  By explicitly
diverging from upstream default install by default, we are effectively
creating an incompatible Python environment and requiring users to go
through extra steps to restore upstream compatibility.

What's worse, Python development is often done via virtual environments.
For regular Python packages, all kinds of Gentoo divergence can be
easily "reverted" by installing the upstream packages inside the virtual
environment.  However, this isn't the case for dev-lang/python itself --
by stripping it down we're effectively creating virtual environments
that are not fully functional and whose functionality cannot be restored
without actually rebuilding dev-lang/python.

On top of that, as you are also perfectly aware, stripping down tests is
often non-trivial and either requires complex patches (we are talking of
modifying the build system not to install something), or ugly hacks (we
are talking about having Python install and compile modules, and
removing them afterwards).

Finally, "requiring other files to be present" effectively means adding
USE dependencies.  This is going to be a lot of effort, considering that
effectively developers would have to run without test suites installed
to detect missing dependencies, and then rebuild dev-lang/python every
time they are about to run tests in a package that actually requires
these files.

Overall, I don't believe you've provided sufficient rationale to justify
diverging from upstream, adding significant complexity to ebuilds
and adding a lot of additional work to package maintainers who will be
responsible for ensuring the correctness of package installations
and USE dependencies.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [gentoo-dev] RFC: Block ebuilds installing tests to ${D} by default

[Eli Schwartz]

[-- Attachment #1.1.1: Type: text/plain, Size: 3164 bytes --]

On 2/1/24 4:03 PM, Michał Górny wrote:
> I suppose you are referring to dev-lang/python here.  Unfortunately,
> removing tests from it is a non-trivial problem.  As I've mentioned to
> you before, there are packages that actually import modules form the
> test directory.
> 
> Remember that Gentoo has a policy of following upstream, and this policy
> is specifically targeted towards developers who expect Gentoo to
> be a good baseline environment for developing packages.  By explicitly
> diverging from upstream default install by default, we are effectively
> creating an incompatible Python environment and requiring users to go
> through extra steps to restore upstream compatibility.


The "following upstream" argument is extremely weak.

eschwartz ~/git/cpython $ ./configure --help| grep test
  --disable-test-modules  don't build nor install test modules


Clearly, upstream is giving you loud and clear permission to refrain
from installing this. They even have a split in their Makefile variables:
- LIBSUBDIRS: things that are always installed
- TESTSUBDIRS: test files that you can disable


> What's worse, Python development is often done via virtual environments.
> For regular Python packages, all kinds of Gentoo divergence can be
> easily "reverted" by installing the upstream packages inside the virtual
> environment.  However, this isn't the case for dev-lang/python itself --
> by stripping it down we're effectively creating virtual environments
> that are not fully functional and whose functionality cannot be restored
> without actually rebuilding dev-lang/python.


This... I sort of hear the argument, and at the same time I kind of do not.

https://docs.python.org/3/library/test.html


"""
Note

The test package is meant for internal use by Python only. It is
documented for the benefit of the core developers of Python. Any use of
this package outside of Python’s standard library is discouraged as code
mentioned here can change or be removed without notice between releases
of Python.
"""

Any users who are developing packages on Gentoo and finding that a
USE=pythontests choice is producing a "not fully functional" virtual
environment which requires rebuilding dev-lang/python to fix, would be
better served obeying the pleading and impassioned requests from
upstream CPython and NOT using the `test` module. Not even `test.support`.

However there are definitely packages in the wild that ignore this, and
require the "test" module.


> This is going to be a lot of effort, considering that
> effectively developers would have to run without test suites installed
> to detect missing dependencies, and then rebuild dev-lang/python every
> time they are about to run tests in a package that actually requires
> these files.


This I think is the biggest issue. It's a relatively simple problem to
solve for any distro that supports a single package recipe producing
multiple packages -- but portage does not (yet!) support one ebuild that
produces two gpkg.tar files, one per "component" of the full set of
installed files.


-- 
Eli Schwartz

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 18399 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] RFC: Block ebuilds installing tests to ${D} by default

[Mike Gilbert]

On Thu, Feb 1, 2024 at 4:38 PM Eli Schwartz <eschwartz93@gmail.com> wrote:
>
> On 2/1/24 4:03 PM, Michał Górny wrote:
> > I suppose you are referring to dev-lang/python here.  Unfortunately,
> > removing tests from it is a non-trivial problem.  As I've mentioned to
> > you before, there are packages that actually import modules form the
> > test directory.
> >
> > Remember that Gentoo has a policy of following upstream, and this policy
> > is specifically targeted towards developers who expect Gentoo to
> > be a good baseline environment for developing packages.  By explicitly
> > diverging from upstream default install by default, we are effectively
> > creating an incompatible Python environment and requiring users to go
> > through extra steps to restore upstream compatibility.
>
>
> The "following upstream" argument is extremely weak.
>
> eschwartz ~/git/cpython $ ./configure --help| grep test
>   --disable-test-modules  don't build nor install test modules
>
>
> Clearly, upstream is giving you loud and clear permission to refrain
> from installing this. They even have a split in their Makefile variables:
> - LIBSUBDIRS: things that are always installed
> - TESTSUBDIRS: test files that you can disable

Ah, that's a relatively new configure option added in Python 3.10. It
didn't exist back in 2015 when I was looking into this.

https://docs.python.org/3/whatsnew/3.10.html#build-changes

Thanks for pointing it out.

Re: [gentoo-dev] RFC: Block ebuilds installing tests to ${D} by default

[Eli Schwartz]

[-- Attachment #1.1.1: Type: text/plain, Size: 1041 bytes --]

On 2/1/24 4:52 PM, Mike Gilbert wrote:
> Ah, that's a relatively new configure option added in Python 3.10. It
> didn't exist back in 2015 when I was looking into this.
> 
> https://docs.python.org/3/whatsnew/3.10.html#build-changes
> 
> Thanks for pointing it out.


Yes -- but personally, to me, it's less about the option existing in
order to avoid having to manually implement it, and more about the fact
that it's a clear message that you have the moral right to implement it
yourself for 3.9 as well.


And the public documentation for the "test" module has told people that
it is purely internal, unstable, liable to change at the drop of a hat,
and documented purely for the benefit of people hacking on the
cpython.git repository... since python 2.6 and earlier Which reinforces
the point that any packages that do `import test.support` are doing
something naughty and we don't need to worry about people complaining
that Gentoo isn't a good upstream-compatible development environment.


-- 
Eli Schwartz

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 18399 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] RFC: Block ebuilds installing tests to ${D} by default

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

Am Donnerstag, 1. Februar 2024, 09:15:39 CET schrieb Robin H. Johnson:
> TL;DR:
> I'd like to propose a change where packages should NOT install their
> tests to ${D} by default. Such an install may optionally enabled with
> USE=test, which should be decoupled from FEATURES=test. Or depending on
> the color of the bikeshed, we add something new like USE=install-tests.

I see where you come from, but the decision what precisely to install
(when we do not follow upstream) can be very non-trivial.

I'm not familiar with Python, but for Perl there is quite some test
infrastructure in the main Perl package... Then there are regular
Perl packages that are extensions to the test suite. These would require
the test modules from core Perl then? I really wouldnt want to figure 
out what to keep and what to drop, and spend a lot of effort getting
the dependencies right.

Also this is an infinite source of upstream "It's Gentoo, we don't
support that because they do weird stuff" bugs.

tl;dr: no



-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer 
(council, comrel, toolchain, base-system, perl, libreoffice)
https://wiki.gentoo.org/wiki/User:Dilfridge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]