public inbox for gentoo-dev@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-dev] Why adding python3_8 to Gentoo sucks?
@ 2019-11-13 21:16 Michał Górny
  2019-11-15  8:05 ` Michał Górny
                   ` (2 more replies)
  0 siblings, 3 replies; 18+ messages in thread
From: Michał Górny @ 2019-11-13 21:16 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 2607 bytes --]

Hi,

I'd like to share my frustration at the state of Python in general,
and Python packages in Gentoo.  So I'd like to 'bootstrap' python3_8 --
that is, add it to the most common dependency, dev-python/setuptools. 
Simple thing, right?


1. There's no such thing as trivial dependency graph in Python.  If you
think that setuptools has a few deps, you're entirely wrong.  I actually
had to write a tool to even assemble list of deps to start with,
and the number is: 174.  I mean, in order to enable py3.8 on setuptools,
you have to enable it on at least 173 other packages.

Sure, some of those packages are just doc-deps or test-deps, and some
could be avoided one way or another.  However, avoiding them is only
temporary and involves more effort than it saves.


2. There are some packages that dropped Python 2.7 but still have 2.7
deps.  So we need to also add py3.8 to older versions that still has
2.7.  Plus, some packages have explicit <-deps.  So we need to add 3.8
to them, and hope that the old version will actually work with 3.8,
and then to their extra dependencies.

This is all handiwork.  The number is now 178 packages, or 187 ebuilds.


3. Of course there are packages with new deps dropping keywords whose
maintainers (or bumpers) never bothered filing a keywordreq.  Because
why bother, somebody else will do that when it blocks everything,
right?!

Well, guess what.  python3_8 flag is going to be masked on non-amd64
because people didn't bother keywording new versions of their packages
on other arches.

If you choose to realize your mistake now, and are willing to fix it,
start keywording new versions.


Here's the initial CI run:

https://github.com/gentoo/gentoo/pull/13638

The packages haven't been tested yet.  If you want to help, feel free to
apply it locally, and run tests in all those packages, and try to
assemble a reasonably readable report of what fails.  Then probably diff
the failures against py3.7 because some packages probably fail there
as well.

Of course many of those packages don't have tests at all.  Because it
was too much effort, and the four-letter company didn't pay for them. 
Because it was too hard to use GitHub snapshot over pypi tarball that
doesn't bundle tests because obviously nobody wants them.  Help with
that welcome too.

Help with getting rid of py2 revdeps of py3-only packages would be very
welcome too.

In other words, there's a lot of work to get Python near-sane in Gentoo,
and we'd welcome all the help we can get.  TIA.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-13 21:16 [gentoo-dev] Why adding python3_8 to Gentoo sucks? Michał Górny
@ 2019-11-15  8:05 ` Michał Górny
  2019-11-15 14:18   ` Rich Freeman
  2019-11-16 11:15   ` Michał Górny
  2019-11-15  8:41 ` Alexey 'Alexxy' Shvetsov
  2019-11-15 15:27 ` Ian Stakenvicius
  2 siblings, 2 replies; 18+ messages in thread
From: Michał Górny @ 2019-11-15  8:05 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

On Wed, 2019-11-13 at 22:16 +0100, Michał Górny wrote:
> I'd like to share my frustration at the state of Python in general,
> and Python packages in Gentoo.  So I'd like to 'bootstrap' python3_8 --
> that is, add it to the most common dependency, dev-python/setuptools. 
> Simple thing, right?
> 

Well, it turned out that things are worse than I anticipated.
I expected *some* minor test breakage.  What we have instead is packages
being so broken that they break a lot of other packages.

So I went with plan B instead: I'll do as much testing locally
as possible, and add py3.8 when I manage to get the tests on the package
in question working, independently of the testing of all deep test deps.
This will mean that some packages will have tests disabled temporarily
for end users.

This means we've got dev-python/setuptools with py3.8 now, and we can
start slowly testing everything.  As the depgraph grows, we'll be able
to reenable tests everywhere.

I will also probably start removing either complete py2 support, or some
of its features (e.g. docs) whenever it causes trouble.  Given that both
dev-python/sphinx (primary doc building tool) and dev-python/pytest (one
of the most common test runners) no longer support py2, we're in a big
mess.


-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-13 21:16 [gentoo-dev] Why adding python3_8 to Gentoo sucks? Michał Górny
  2019-11-15  8:05 ` Michał Górny
@ 2019-11-15  8:41 ` Alexey 'Alexxy' Shvetsov
  2019-11-15  8:45   ` Michał Górny
  2019-11-15 15:27 ` Ian Stakenvicius
  2 siblings, 1 reply; 18+ messages in thread
From: Alexey 'Alexxy' Shvetsov @ 2019-11-15  8:41 UTC (permalink / raw
  To: gentoo-dev; +Cc: Michał Górny

[-- Attachment #1: Type: text/plain, Size: 3256 bytes --]

I think problem is more global: 
* many python3_6 packages dont have python3_7 keywords, because its 
maintainers dont bother about it. So if you want to switch to python3_7 you 
still need manualy add python3_7 use for  many packages (that actualy work 
without problems
* we need policy for python packages that force enablement of new python 
version on existing packages.

В письме от четверг, 14 ноября 2019 г. 00:16:10 MSK пользователь Michał Górny 
написал:
> Hi,
> 
> I'd like to share my frustration at the state of Python in general,
> and Python packages in Gentoo.  So I'd like to 'bootstrap' python3_8 --
> that is, add it to the most common dependency, dev-python/setuptools.
> Simple thing, right?
> 
> 
> 1. There's no such thing as trivial dependency graph in Python.  If you
> think that setuptools has a few deps, you're entirely wrong.  I actually
> had to write a tool to even assemble list of deps to start with,
> and the number is: 174.  I mean, in order to enable py3.8 on setuptools,
> you have to enable it on at least 173 other packages.
> 
> Sure, some of those packages are just doc-deps or test-deps, and some
> could be avoided one way or another.  However, avoiding them is only
> temporary and involves more effort than it saves.
> 
> 
> 2. There are some packages that dropped Python 2.7 but still have 2.7
> deps.  So we need to also add py3.8 to older versions that still has
> 2.7.  Plus, some packages have explicit <-deps.  So we need to add 3.8
> to them, and hope that the old version will actually work with 3.8,
> and then to their extra dependencies.
> 
> This is all handiwork.  The number is now 178 packages, or 187 ebuilds.
> 
> 
> 3. Of course there are packages with new deps dropping keywords whose
> maintainers (or bumpers) never bothered filing a keywordreq.  Because
> why bother, somebody else will do that when it blocks everything,
> right?!
> 
> Well, guess what.  python3_8 flag is going to be masked on non-amd64
> because people didn't bother keywording new versions of their packages
> on other arches.
> 
> If you choose to realize your mistake now, and are willing to fix it,
> start keywording new versions.
> 
> 
> Here's the initial CI run:
> 
> https://github.com/gentoo/gentoo/pull/13638
> 
> The packages haven't been tested yet.  If you want to help, feel free to
> apply it locally, and run tests in all those packages, and try to
> assemble a reasonably readable report of what fails.  Then probably diff
> the failures against py3.7 because some packages probably fail there
> as well.
> 
> Of course many of those packages don't have tests at all.  Because it
> was too much effort, and the four-letter company didn't pay for them.
> Because it was too hard to use GitHub snapshot over pypi tarball that
> doesn't bundle tests because obviously nobody wants them.  Help with
> that welcome too.
> 
> Help with getting rid of py2 revdeps of py3-only packages would be very
> welcome too.
> 
> In other words, there's a lot of work to get Python near-sane in Gentoo,
> and we'd welcome all the help we can get.  TIA.


-- 
Best regards,
Alexey 'Alexxy' Shvetsov

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15  8:41 ` Alexey 'Alexxy' Shvetsov
@ 2019-11-15  8:45   ` Michał Górny
  2019-11-15 10:20     ` Alexey 'Alexxy' Shvetsov
  0 siblings, 1 reply; 18+ messages in thread
From: Michał Górny @ 2019-11-15  8:45 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 639 bytes --]

On Fri, 2019-11-15 at 11:41 +0300, Alexey 'Alexxy' Shvetsov wrote:
> I think problem is more global: 
> * many python3_6 packages dont have python3_7 keywords, because its 
> maintainers dont bother about it. So if you want to switch to python3_7 you 
> still need manualy add python3_7 use for  many packages (that actualy work 
> without problems
> * we need policy for python packages that force enablement of new python 
> version on existing packages.

Policy makes little sense if there's no way to enforce it.

If you tested some package on py3.7 and depgraph, just add it there.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15  8:45   ` Michał Górny
@ 2019-11-15 10:20     ` Alexey 'Alexxy' Shvetsov
  2019-11-15 10:47       ` Mart Raudsepp
  0 siblings, 1 reply; 18+ messages in thread
From: Alexey 'Alexxy' Shvetsov @ 2019-11-15 10:20 UTC (permalink / raw
  To: gentoo-dev; +Cc: Michał Górny

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

В письме от пятница, 15 ноября 2019 г. 11:45:32 MSK пользователь Michał Górny 
написал:
> On Fri, 2019-11-15 at 11:41 +0300, Alexey 'Alexxy' Shvetsov wrote:
> > I think problem is more global:
> > * many python3_6 packages dont have python3_7 keywords, because its
> > maintainers dont bother about it. So if you want to switch to python3_7
> > you
> > still need manualy add python3_7 use for  many packages (that actualy work
> > without problems
> > * we need policy for python packages that force enablement of new python
> > version on existing packages.
> 
> Policy makes little sense if there's no way to enforce it.
> 
> If you tested some package on py3.7 and depgraph, just add it there.

Some people dont like it =D And some arches (doesnt matter if i have such hw 
and tested on it)


-- 
Best regards,
Alexey 'Alexxy' Shvetsov

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15 10:20     ` Alexey 'Alexxy' Shvetsov
@ 2019-11-15 10:47       ` Mart Raudsepp
  2019-11-15 12:04         ` Alexey 'Alexxy' Shvetsov
  0 siblings, 1 reply; 18+ messages in thread
From: Mart Raudsepp @ 2019-11-15 10:47 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1349 bytes --]

Ühel kenal päeval, R, 15.11.2019 kell 13:20, kirjutas Alexey 'Alexxy'
Shvetsov:
> В письме от пятница, 15 ноября 2019 г. 11:45:32 MSK пользователь
> Michał Górny 
> написал:
> > On Fri, 2019-11-15 at 11:41 +0300, Alexey 'Alexxy' Shvetsov wrote:
> > > I think problem is more global:
> > > * many python3_6 packages dont have python3_7 keywords, because
> > > its
> > > maintainers dont bother about it. So if you want to switch to
> > > python3_7
> > > you
> > > still need manualy add python3_7 use for  many packages (that
> > > actualy work
> > > without problems
> > > * we need policy for python packages that force enablement of new
> > > python
> > > version on existing packages.
> > 
> > Policy makes little sense if there's no way to enforce it.
> > 
> > If you tested some package on py3.7 and depgraph, just add it
> > there.
> 
> Some people dont like it =D And some arches (doesnt matter if i have
> such hw 
> and tested on it)

People mostly don't like it when you throw unrelated changes on top of
that, under the guise of just python target addition, especially when
they are also broken changes and involve a revbump without maintainers
involvement (just PYTHON_TARGETS addition doesn't). Or if it actually
doesn't work with the added python targets.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15 10:47       ` Mart Raudsepp
@ 2019-11-15 12:04         ` Alexey 'Alexxy' Shvetsov
  2019-11-15 12:19           ` Michael Orlitzky
  2019-11-15 12:22           ` Michał Górny
  0 siblings, 2 replies; 18+ messages in thread
From: Alexey 'Alexxy' Shvetsov @ 2019-11-15 12:04 UTC (permalink / raw
  To: gentoo-dev; +Cc: Mart Raudsepp

[-- Attachment #1: Type: text/plain, Size: 1875 bytes --]

В письме от пятница, 15 ноября 2019 г. 13:47:12 MSK пользователь Mart Raudsepp 
написал:
> Ühel kenal päeval, R, 15.11.2019 kell 13:20, kirjutas Alexey 'Alexxy'
> 
> Shvetsov:
> > В письме от пятница, 15 ноября 2019 г. 11:45:32 MSK пользователь
> > Michał Górny
> > 
> > написал:
> > > On Fri, 2019-11-15 at 11:41 +0300, Alexey 'Alexxy' Shvetsov wrote:
> > > > I think problem is more global:
> > > > * many python3_6 packages dont have python3_7 keywords, because
> > > > its
> > > > maintainers dont bother about it. So if you want to switch to
> > > > python3_7
> > > > you
> > > > still need manualy add python3_7 use for  many packages (that
> > > > actualy work
> > > > without problems
> > > > * we need policy for python packages that force enablement of new
> > > > python
> > > > version on existing packages.
> > > 
> > > Policy makes little sense if there's no way to enforce it.
> > > 
> > > If you tested some package on py3.7 and depgraph, just add it
> > > there.
> > 
> > Some people dont like it =D And some arches (doesnt matter if i have
> > such hw
> > and tested on it)
> 
> People mostly don't like it when you throw unrelated changes on top of
> that, under the guise of just python target addition, especially when
> they are also broken changes and involve a revbump without maintainers
> involvement (just PYTHON_TARGETS addition doesn't). Or if it actually
> doesn't work with the added python targets.

As i remember some decades ago policy was: revbump needed if you change 
chnages stuff installed on filesystem. So in case of py addition it is. So what 
changed? 

Are there some new written rules that says in what case you need revbump and 
in what it needed? 


-- 
Best regards,
Alexey 'Alexxy' Shvetsov

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15 12:04         ` Alexey 'Alexxy' Shvetsov
@ 2019-11-15 12:19           ` Michael Orlitzky
  2019-11-15 13:54             ` Alexey 'Alexxy' Shvetsov
  2019-11-15 12:22           ` Michał Górny
  1 sibling, 1 reply; 18+ messages in thread
From: Michael Orlitzky @ 2019-11-15 12:19 UTC (permalink / raw
  To: gentoo-dev

On 11/15/19 7:04 AM, Alexey 'Alexxy' Shvetsov wrote:
> 
> As i remember some decades ago policy was: revbump needed if you change 
> chnages stuff installed on filesystem. So in case of py addition it is. So what 
> changed? 
> 
> Are there some new written rules that says in what case you need revbump and 
> in what it needed? 
> 

To avoid breaking dependency resolution on users' systems, you have to
make a new revision whenever you change a metadata variable that the
package manager uses: DEPEND, RDEPEND, BDEPEND, IUSE, LICENSE,
PYTHON_COMPAT, etc.

Basically, unless your change is completely trivial (updated comment,
typo fix...), you should be making a new revision by default.


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15 12:04         ` Alexey 'Alexxy' Shvetsov
  2019-11-15 12:19           ` Michael Orlitzky
@ 2019-11-15 12:22           ` Michał Górny
  2019-11-15 12:29             ` Michael Orlitzky
  1 sibling, 1 reply; 18+ messages in thread
From: Michał Górny @ 2019-11-15 12:22 UTC (permalink / raw
  To: gentoo-dev; +Cc: Mart Raudsepp

[-- Attachment #1: Type: text/plain, Size: 2100 bytes --]

On Fri, 2019-11-15 at 15:04 +0300, Alexey 'Alexxy' Shvetsov wrote:
> В письме от пятница, 15 ноября 2019 г. 13:47:12 MSK пользователь Mart Raudsepp 
> написал:
> > Ühel kenal päeval, R, 15.11.2019 kell 13:20, kirjutas Alexey 'Alexxy'
> > 
> > Shvetsov:
> > > В письме от пятница, 15 ноября 2019 г. 11:45:32 MSK пользователь
> > > Michał Górny
> > > 
> > > написал:
> > > > On Fri, 2019-11-15 at 11:41 +0300, Alexey 'Alexxy' Shvetsov wrote:
> > > > > I think problem is more global:
> > > > > * many python3_6 packages dont have python3_7 keywords, because
> > > > > its
> > > > > maintainers dont bother about it. So if you want to switch to
> > > > > python3_7
> > > > > you
> > > > > still need manualy add python3_7 use for  many packages (that
> > > > > actualy work
> > > > > without problems
> > > > > * we need policy for python packages that force enablement of new
> > > > > python
> > > > > version on existing packages.
> > > > 
> > > > Policy makes little sense if there's no way to enforce it.
> > > > 
> > > > If you tested some package on py3.7 and depgraph, just add it
> > > > there.
> > > 
> > > Some people dont like it =D And some arches (doesnt matter if i have
> > > such hw
> > > and tested on it)
> > 
> > People mostly don't like it when you throw unrelated changes on top of
> > that, under the guise of just python target addition, especially when
> > they are also broken changes and involve a revbump without maintainers
> > involvement (just PYTHON_TARGETS addition doesn't). Or if it actually
> > doesn't work with the added python targets.
> 
> As i remember some decades ago policy was: revbump needed if you change 
> chnages stuff installed on filesystem. So in case of py addition it is. So what 
> changed? 
> 
> Are there some new written rules that says in what case you need revbump and 
> in what it needed? 

https://devmanual.gentoo.org/general-concepts/ebuild-revisions/index.html

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15 12:22           ` Michał Górny
@ 2019-11-15 12:29             ` Michael Orlitzky
  2019-11-16 13:55               ` Ulrich Mueller
  0 siblings, 1 reply; 18+ messages in thread
From: Michael Orlitzky @ 2019-11-15 12:29 UTC (permalink / raw
  To: gentoo-dev

On 11/15/19 7:22 AM, Michał Górny wrote:
> 
> https://devmanual.gentoo.org/general-concepts/ebuild-revisions/index.html
> 

Two things on that page are outright wrong:

  1. If the package has stable keywords, they should be moved to the new
     revision without dropping. To commit the ebuild, repoman commit
     --straight-to-stable option should be used.

That violates our other stabilization policies and common sense.

  2. Examples of changes that can be done without a revision bump are...

     adding a new USE flag or removing an existing one (since change in
     USE flags is going to trigger --changed-use rebuild),

The --changed-use flag is not default, not part of the PMS, and slows
portage down even more than it already is. The council ratified the PMS
and the associated GLEP, and following this advice breaks PMS-compliant
package managers.


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15 12:19           ` Michael Orlitzky
@ 2019-11-15 13:54             ` Alexey 'Alexxy' Shvetsov
  0 siblings, 0 replies; 18+ messages in thread
From: Alexey 'Alexxy' Shvetsov @ 2019-11-15 13:54 UTC (permalink / raw
  To: gentoo-dev; +Cc: Michael Orlitzky

[-- Attachment #1: Type: text/plain, Size: 938 bytes --]

В письме от пятница, 15 ноября 2019 г. 15:19:31 MSK пользователь Michael 
Orlitzky написал:
> On 11/15/19 7:04 AM, Alexey 'Alexxy' Shvetsov wrote:
> > As i remember some decades ago policy was: revbump needed if you change
> > chnages stuff installed on filesystem. So in case of py addition it is. So
> > what changed?
> > 
> > Are there some new written rules that says in what case you need revbump
> > and in what it needed?
> 
> To avoid breaking dependency resolution on users' systems, you have to
> make a new revision whenever you change a metadata variable that the
> package manager uses: DEPEND, RDEPEND, BDEPEND, IUSE, LICENSE,
> PYTHON_COMPAT, etc.
> 
> Basically, unless your change is completely trivial (updated comment,
> typo fix...), you should be making a new revision by default.

Yep. That that i remember. 

-- 
Best regards,
Alexey 'Alexxy' Shvetsov

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15  8:05 ` Michał Górny
@ 2019-11-15 14:18   ` Rich Freeman
  2019-11-16 11:15   ` Michał Górny
  1 sibling, 0 replies; 18+ messages in thread
From: Rich Freeman @ 2019-11-15 14:18 UTC (permalink / raw
  To: gentoo-dev

On Fri, Nov 15, 2019 at 3:05 AM Michał Górny <mgorny@gentoo.org> wrote:
>
> On Wed, 2019-11-13 at 22:16 +0100, Michał Górny wrote:
> > I'd like to share my frustration at the state of Python in general,
> > and Python packages in Gentoo.  So I'd like to 'bootstrap' python3_8 --
> > that is, add it to the most common dependency, dev-python/setuptools.
> > Simple thing, right?
> >
>
> So I went with plan B instead: I'll do as much testing locally
> as possible, and add py3.8 when I manage to get the tests on the package
> in question working, independently of the testing of all deep test deps.
> This will mean that some packages will have tests disabled temporarily
> for end users.
>

Perhaps an overlay would be simpler just so that you can generally
avoid worrying about QA until you're tidying up, but otherwise this
seems like it could be done in-tree by just masking the use flag so
that only those willing to test/contribute run into issues.

You've described a number of issues and my sense is that many are just
inherent to python itself (the complex dep graph/etc - unless we want
a monolithic package).  Some of course go to Gentoo practices, some of
which cause pain outside of python.

In particular it seems like many still don't understand when
revbumping is necessary.  I'd have to dig up the wording of the actual
decision but as I recall when the Council made the decision they
wanted to leave a bit of room for maintainer discretion, trusting that
maintainers would use it properly.  An alternative proposal was to
just make a strict rule that would have erred on the side of QA, at
the cost of extra rebuilds for users (but at least consistent ones
that didn't break package managers).  Obviously developers can't
exercise proper discretion if they don't fully understand the impacts.
If in doubt a revbump should always be safe, just at the cost of some
rebuilds (which are probably cheap for python packages).

-- 
Rich


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-13 21:16 [gentoo-dev] Why adding python3_8 to Gentoo sucks? Michał Górny
  2019-11-15  8:05 ` Michał Górny
  2019-11-15  8:41 ` Alexey 'Alexxy' Shvetsov
@ 2019-11-15 15:27 ` Ian Stakenvicius
  2 siblings, 0 replies; 18+ messages in thread
From: Ian Stakenvicius @ 2019-11-15 15:27 UTC (permalink / raw
  To: gentoo-dev

On 2019-11-13 4:16 p.m., Michał Górny wrote:
> [ Snip! ]

Can automation help with this at all, or is automation being used 
already?

Also to clarify, is the issue here the way that we specify python 
versions syntactically in all of the ebuilds in the repo, or is it 
just the way the dependency graph and compatibility of packages 
against a new python variant pan out?



^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15  8:05 ` Michał Górny
  2019-11-15 14:18   ` Rich Freeman
@ 2019-11-16 11:15   ` Michał Górny
  1 sibling, 0 replies; 18+ messages in thread
From: Michał Górny @ 2019-11-16 11:15 UTC (permalink / raw
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1973 bytes --]

On Fri, 2019-11-15 at 09:05 +0100, Michał Górny wrote:
> On Wed, 2019-11-13 at 22:16 +0100, Michał Górny wrote:
> > I'd like to share my frustration at the state of Python in general,
> > and Python packages in Gentoo.  So I'd like to 'bootstrap' python3_8 --
> > that is, add it to the most common dependency, dev-python/setuptools. 
> > Simple thing, right?
> > 
> 
> Well, it turned out that things are worse than I anticipated.
> I expected *some* minor test breakage.  What we have instead is packages
> being so broken that they break a lot of other packages.
> 
> So I went with plan B instead: I'll do as much testing locally
> as possible, and add py3.8 when I manage to get the tests on the package
> in question working, independently of the testing of all deep test deps.
> This will mean that some packages will have tests disabled temporarily
> for end users.
> 

Good news, everyone.  We have 3.8-enabled setuptools, pytest and nose
which should be sufficient to start testing other packages.

Of other common packages, dev-python/sphinx isn't 3.8-enabled yet. 
However, this gives a good exercise of improving sphinx deps.  That is:

1. If package uses sphinx only to build docs from .rst files, has no
extra deps and doesn't use code introspection, you can just drop
PYTHON_USEDEP.

2. Likewise but if it has extra deps, you can use the very hard any-r1
API in python-r1 (distutils-r1).  This has the advantage that it uses
any-of deps, and doesn't require matching PYTHON_TARGETS.  Example:

BDEPEND="
  doc? ( $(python_gen_any_dep '
    dev-python/sphinx[${PYTHON_USEDEP}]
    dev-python/sphinx-some-random-theme[${PYTHON_USEDEP}]
  ')"

python_check_deps() {
  use doc || return 0
  has_version "dev-python/sphinx[${PYTHON_USEDEP}]" &&
    has_version "dev-python/sphinx-some-random-theme[${PYTHON_USEDEP}]"
}

python_compile_all {
  use doc && usual_stuff
}


-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-15 12:29             ` Michael Orlitzky
@ 2019-11-16 13:55               ` Ulrich Mueller
  2019-11-16 21:41                 ` Michał Górny
  2019-11-17 17:07                 ` Michael Orlitzky
  0 siblings, 2 replies; 18+ messages in thread
From: Ulrich Mueller @ 2019-11-16 13:55 UTC (permalink / raw
  To: Michael Orlitzky; +Cc: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

>>>>> On Fri, 15 Nov 2019, Michael Orlitzky wrote:

> Two things on that page are outright wrong:

>   1. If the package has stable keywords, they should be moved to the
>      new revision without dropping. To commit the ebuild, repoman
>      commit --straight-to-stable option should be used.

> That violates our other stabilization policies and common sense.

This is in the context of revision bumps, not version bumps. Also, the
item immediately preceding it clarifies that any non-trivial changes
require dropping to ~arch.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-16 13:55               ` Ulrich Mueller
@ 2019-11-16 21:41                 ` Michał Górny
  2019-11-17 17:15                   ` Michael Orlitzky
  2019-11-17 17:07                 ` Michael Orlitzky
  1 sibling, 1 reply; 18+ messages in thread
From: Michał Górny @ 2019-11-16 21:41 UTC (permalink / raw
  To: gentoo-dev, Michael Orlitzky

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

On Sat, 2019-11-16 at 14:55 +0100, Ulrich Mueller wrote:
> > > > > > On Fri, 15 Nov 2019, Michael Orlitzky wrote:
> > Two things on that page are outright wrong:
> >   1. If the package has stable keywords, they should be moved to the
> >      new revision without dropping. To commit the ebuild, repoman
> >      commit --straight-to-stable option should be used.
> > That violates our other stabilization policies and common sense.
> 
> This is in the context of revision bumps, not version bumps. Also, the
> item immediately preceding it clarifies that any non-trivial changes
> require dropping to ~arch.
> 

More precisely, this is in context of dependency corrections.  There is
no need to go through stabilization to restrict too broad dependency
specifications, while stable users hit the issue for the next two
months.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-16 13:55               ` Ulrich Mueller
  2019-11-16 21:41                 ` Michał Górny
@ 2019-11-17 17:07                 ` Michael Orlitzky
  1 sibling, 0 replies; 18+ messages in thread
From: Michael Orlitzky @ 2019-11-17 17:07 UTC (permalink / raw
  To: gentoo-dev

On 11/16/19 8:55 AM, Ulrich Mueller wrote:
> 
> This is in the context of revision bumps, not version bumps.

The things allowed in a revision bump are,

  * New EAPI,
  * Adding/dropping eclasses,
  * Adding/dropping dependencies,
  * Adding/dropping a phase function,
  * Literally anything, except changing SRC_URI to a new version.

Everything that can go wrong in a new version can go wrong in a new
revision, and almost none of them should go straight to stable.


> Also, the item immediately preceding it clarifies that any
> non-trivial changes require dropping to ~arch.

I don't think it reads that way. The bullet list consists of
not-mutually-exclusive items, and it's not clear if the items are to be
"and"ed or "or"ed in the first place.


^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [gentoo-dev] Why adding python3_8 to Gentoo sucks?
  2019-11-16 21:41                 ` Michał Górny
@ 2019-11-17 17:15                   ` Michael Orlitzky
  0 siblings, 0 replies; 18+ messages in thread
From: Michael Orlitzky @ 2019-11-17 17:15 UTC (permalink / raw
  To: gentoo-dev

On 11/16/19 4:41 PM, Michał Górny wrote:
> 
> More precisely, this is in context of dependency corrections.  There is
> no need to go through stabilization to restrict too broad dependency
> specifications, while stable users hit the issue for the next two
> months.
> 

The word "dependency" doesn't appear on that page before the line that I
have a problem with.

I'm not arguing against common sense: if you need to fix something
that's completely broken in a stable ebuild and if that fix requires a
new revision, then do a new (straight to stable) revision. However,
that's a rare situation, and the bullet point doesn't make it clear that
it's referring to a specific rare situation that should be ignored 99%
of the time in favor of the first bullet point.

To make matters worse, the fact that you can push commits
straight-to-stable to fix a bad issue in the stable tree is completely
independent of whether or not you make a new revision. You could push an
entirely new version with the same goal. So the fact that the exception
to the rule appears as a bullet point on the "ebuild revisions" page
only sows further confusion.

When people pull up that page, that want a simple heuristic to follow,
not a legal document that they have to decode for half an hour before
they can fix a bug.


^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2019-11-17 17:15 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-11-13 21:16 [gentoo-dev] Why adding python3_8 to Gentoo sucks? Michał Górny
2019-11-15  8:05 ` Michał Górny
2019-11-15 14:18   ` Rich Freeman
2019-11-16 11:15   ` Michał Górny
2019-11-15  8:41 ` Alexey 'Alexxy' Shvetsov
2019-11-15  8:45   ` Michał Górny
2019-11-15 10:20     ` Alexey 'Alexxy' Shvetsov
2019-11-15 10:47       ` Mart Raudsepp
2019-11-15 12:04         ` Alexey 'Alexxy' Shvetsov
2019-11-15 12:19           ` Michael Orlitzky
2019-11-15 13:54             ` Alexey 'Alexxy' Shvetsov
2019-11-15 12:22           ` Michał Górny
2019-11-15 12:29             ` Michael Orlitzky
2019-11-16 13:55               ` Ulrich Mueller
2019-11-16 21:41                 ` Michał Górny
2019-11-17 17:15                   ` Michael Orlitzky
2019-11-17 17:07                 ` Michael Orlitzky
2019-11-15 15:27 ` Ian Stakenvicius

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox