[gentoo-user] 17 new packages because pip wants to write poetry?

[gentoo-user] 17 new packages because pip wants to write poetry?

[Grant Edwards]

This morning a routine emerge -auvND wanted to install 17 new packages
for no apparent reason.

Adding a 't' to the emerge options seems to point to pip, which now
wants to install a whole shed-load of new packages — among them
dev-python/poetry and a bunch of markdown and rich-text libraries. Oh
great, pip has incorporated AI, become self-aware, and wants to write
angst-ridden poetry about the futility of "life".

It turns out dev-python/poetry has nothing to do with poetry, so my AI
paranoia was unjustified (this time), but one wonders what devs are
thinking when the decide they add dozens of new dependencies like
that. Why does pip suddenly need to format (or produce?) both markdown
and RTF when it's been able to get along fine without them for so many
years?

--
Grant

Re: [gentoo-user] 17 new packages because pip wants to write poetry?

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1340 bytes --]

On 7/29/24 12:01 PM, Grant Edwards wrote:
> This morning a routine emerge -auvND wanted to install 17 new packages
> for no apparent reason.
> 
> Adding a 't' to the emerge options seems to point to pip, which now
> wants to install a whole shed-load of new packages — among them
> dev-python/poetry and a bunch of markdown and rich-text libraries. Oh
> great, pip has incorporated AI, become self-aware, and wants to write
> angst-ridden poetry about the futility of "life".
> 
> It turns out dev-python/poetry has nothing to do with poetry, so my AI
> paranoia was unjustified (this time), but one wonders what devs are
> thinking when the decide they add dozens of new dependencies like
> that. Why does pip suddenly need to format (or produce?) both markdown
> and RTF when it's been able to get along fine without them for so many
> years?

For many years, pip has contained bundled libraries. These libraries
recently got unbundled, and now you're installing a system copy.

pip has always "needed to format (or produce?) both markdown and RTF",
but it also "needs" to use bundled libraries to do it without people
noticing that it does it.

Some of those packages are only bdeps, and you can feel free to e.g.
delete poetry via emerge -c --with-bdeps=n once you're done updating.


-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[gentoo-user] Re: 17 new packages because pip wants to write poetry?

[Grant Edwards]

On 2024-07-29, Eli Schwartz <eschwartz@gentoo.org> wrote:
>
>> It turns out dev-python/poetry has nothing to do with poetry, so my AI
>> paranoia was unjustified (this time), but one wonders what devs are
>> thinking when the decide they add dozens of new dependencies like
>> that. Why does pip suddenly need to format (or produce?) both markdown
>> and RTF when it's been able to get along fine without them for so many
>> years?
>
> For many years, pip has contained bundled libraries. These libraries
> recently got unbundled, and now you're installing a system copy.

Yep, I figured that was probably the answer.  I (for one) would
appreciate some sort of notice when such an unbundling happens so that
I don't waste time trying to track down why emerge suddenly wants to
install a bunch of new packages.  I can't really come up with a good
mechanism for that other than news items.

> pip has always "needed to format (or produce?) both markdown and RTF",

OK, now I'm genuinely curious: what does pip need to do with markdown
and RTF? My first guess was that its man pages aren't written in nroff
any more and somehow markdown was being used. [I already had at least
one markdown implementation installed, but pip apparently demands a
different one.] But there is no man page for pip. There are a bunch of
documents in /usr/share/doc/pip-<ver>, but they're all reStructured Text.

Oh well, I guess I should be thankful it didn't force LaTeX and pandoc
installs.

> but it also "needs" to use bundled libraries to do it without people
> noticing that it does it.  Some of those packages are only bdeps,
> and you can feel free to e.g. delete poetry via
> emerge -c --with-bdeps=n once you're done updating.

I usually just leave bdeps installed. Otherwise that would remove 130
other packages as well (some of which take a looong time to build).

--
Grant

Re: [gentoo-user] Re: 17 new packages because pip wants to write poetry?

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 2963 bytes --]

On 7/29/24 2:05 PM, Grant Edwards wrote:
> On 2024-07-29, Eli Schwartz <eschwartz@gentoo.org> wrote:
>>
>>> It turns out dev-python/poetry has nothing to do with poetry, so my AI
>>> paranoia was unjustified (this time), but one wonders what devs are
>>> thinking when the decide they add dozens of new dependencies like
>>> that. Why does pip suddenly need to format (or produce?) both markdown
>>> and RTF when it's been able to get along fine without them for so many
>>> years?
>>
>> For many years, pip has contained bundled libraries. These libraries
>> recently got unbundled, and now you're installing a system copy.
> 
> Yep, I figured that was probably the answer.  I (for one) would
> appreciate some sort of notice when such an unbundling happens so that
> I don't waste time trying to track down why emerge suddenly wants to
> install a bunch of new packages.  I can't really come up with a good
> mechanism for that other than news items.


Well, it is there in the `git log` of the package. And at
https://packages.gentoo.org/packages/dev-python/pip/changelog


 Commits on 2024-06-23

dev-python/pip: Unbundle dependencies


>> pip has always "needed to format (or produce?) both markdown and RTF",
> 
> OK, now I'm genuinely curious: what does pip need to do with markdown
> and RTF? My first guess was that its man pages aren't written in nroff
> any more and somehow markdown was being used. [I already had at least
> one markdown implementation installed, but pip apparently demands a
> different one.] But there is no man page for pip. There are a bunch of
> documents in /usr/share/doc/pip-<ver>, but they're all reStructured Text.
> 
> Oh well, I guess I should be thankful it didn't force LaTeX and pandoc
> installs.


Python software loves to depend on python software and hates to depend
on anything that isn't written in python, so I think you're pretty safe
there. :)

As for why it needs to format markdown -- build dependencies of python
software often do, since they want to render the darned stuff into the
https://pypi.org display page for that software and the one and only way
to do so is to render it into the metadata files which get installed at
runtime -- and which are also directly displayed on https://pypi.org

I don't understand it myself, either.

In this case it's actually a bit worse because pip internally uses, and
usually bundles a code copy, of https://pypi.org/project/rich/

"rich" can do a lot of things, including take markdown and print it with
fancy formatting and colors to your terminal emulator. pip isn't using
most of the features of rich, but it is *using* rich at all, and
therefore markdown ends up as a recursive dependency.


...

aside: there are pip manpages, funny you should mention that.

I could totally add another bdepend on sphinx for this! But I would have
to package some things first. :(


-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

[gentoo-user] Re: 17 new packages because pip wants to write poetry?

[Grant Edwards]

On 2024-07-29, Eli Schwartz <eschwartz@gentoo.org> wrote:

>> [...] I (for one) would appreciate some sort of notice when such an
>> unbundling happens so that I don't waste time trying to track down
>> why emerge suddenly wants to install a bunch of new packages.  I
>> can't really come up with a good mechanism for that other than news
>> items.
>
> Well, it is there in the `git log` of the package. And at
> https://packages.gentoo.org/packages/dev-python/pip/changelog
>
>  Commits on 2024-06-23
>
> dev-python/pip: Unbundle dependencies

Right, but that's only useful after you track down the trigger for the
new packages. What would be nice is avoiding that "tracking down"
effort.  [I know, I should just relax, hit 'Y', and trust that emerge
and the devs know what they're doing.]

>> OK, now I'm genuinely curious: what does pip need to do with markdown
>> and RTF? My first guess was that its man pages aren't written in nroff
>> any more and somehow markdown was being used. [I already had at least
>> one markdown implementation installed, but pip apparently demands a
>> different one.] But there is no man page for pip. There are a bunch of
>> documents in /usr/share/doc/pip-<ver>, but they're all reStructured Text.
>> 
>> Oh well, I guess I should be thankful it didn't force LaTeX and pandoc
>> installs.
>
> Python software loves to depend on python software and hates to depend
> on anything that isn't written in python, so I think you're pretty safe
> there. :)

Good point. :)

> As for why it needs to format markdown -- build dependencies of
> python software often do, since they want to render the darned stuff
> into the https://pypi.org display page for that software

Oh. Is that display page (in html?) written into a log somewhere or
shown during the build?

> and the one and only way to do so is to render it into the metadata
> files which get installed at runtime -- and which are also directly
> displayed on https://pypi.org
>
> I don't understand it myself, either.

You certainly understand more of it than I do.

> In this case it's actually a bit worse because pip internally uses, and
> usually bundles a code copy, of https://pypi.org/project/rich/
>
> "rich" can do a lot of things, including take markdown and print it with
> fancy formatting and colors to your terminal emulator. pip isn't using
> most of the features of rich, but it is *using* rich at all, and
> therefore markdown ends up as a recursive dependency.
> ...
>
> aside: there are pip manpages, funny you should mention that.

When installed on Gentoo using dev-python/pip?

> I could totally add another bdepend on sphinx for this! But I would have
> to package some things first. :(

No thanks, sphinx would pull in 10 more packages. :)

If I need pip documentation, I can google for it or look at the
rst.bz2 files install in /usr/share/doc...

Thanks for tolerating my whinging.

--
Grant

Re: [gentoo-user] Re: 17 new packages because pip wants to write poetry?

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 2365 bytes --]

On 7/29/24 4:09 PM, Grant Edwards wrote:
> Right, but that's only useful after you track down the trigger for the
> new packages. What would be nice is avoiding that "tracking down"
> effort.  [I know, I should just relax, hit 'Y', and trust that emerge
> and the devs know what they're doing.]


That's basically just emerge -t though?


>> As for why it needs to format markdown -- build dependencies of
>> python software often do, since they want to render the darned stuff
>> into the https://pypi.org display page for that software
> 
> Oh. Is that display page (in html?) written into a log somewhere or
> shown during the build?


It's stored inside an installed file called

/usr/lib/python3.XX/site-packages/${PN}-${PV}.dist-info/METADATA

The file format is described at
https://packaging.python.org/en/latest/specifications/core-metadata/#description

Of course, the specific metadata field in question is not actually
useful for installed packages, only for web repository uploads, but
there you have it...

There is some useful metadata in that file, for example pip needs it to
be able to list which names and versions are installed, and also to
check whether each python module has its dependencies (or Extras
dependencies) installed. But the stuff that uses rst and markdown is
totally unnecessary for this, even though python build systems have to
add it.


>> aside: there are pip manpages, funny you should mention that.
> 
> When installed on Gentoo using dev-python/pip?
> 
>> I could totally add another bdepend on sphinx for this! But I would have
>> to package some things first. :(
> 
> No thanks, sphinx would pull in 10 more packages. :)
> 
> If I need pip documentation, I can google for it or look at the
> rst.bz2 files install in /usr/share/doc...
> 
> Thanks for tolerating my whinging.


Gentoo doesn't install the manpages, no.

We should. And per policy, manpages cannot be disabled by a USE flag:
https://projects.gentoo.org/qa/policy-guide/installed-files.html#pg0305

But what we can do is build the manpages ourselves and add an extra
SRC_URI to download that. And in this case it's a royal pain to package,
including the fact that it requires sphinxcontrib-towncrier which has
never released any version that isn't an alpha... amazing...


-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-user] 17 new packages because pip wants to write poetry?

[Andreas Fink]

On Mon, 29 Jul 2024 16:01:19 -0000 (UTC)
Grant Edwards <grant.b.edwards@gmail.com> wrote:

> It turns out dev-python/poetry has nothing to do with poetry, so my AI
> paranoia was unjustified (this time), but one wonders what devs are
> thinking when the decide they add dozens of new dependencies like
> that.

Devs are thinking how to make their life easier and not how to please
you.
pip is opensource, go ahead, unbundle the unnecessary bits, help the
devs making it work also without these dependencies.
I am in no way involved in the development of pip, but this attitude is
very counterproductive.