[gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Patrick Lauer]

tl;dr: We should use EAPI5 features

I've noticed some libraries (e.g. poppler) having (almost) all their
consumers migrated to eapi5 subslots.
So upgrading those is now really neato.

Other libraries are still a bit less optimal. So there's lots of
revdep-rebuild / emerge @preserved-rebuild happening, which is annoying.

Thus I suggest declaring a policy:

""
Any library bump that would trigger revdep-rebuild should be done with
the affected library package.mask'ed until all its consumers have been
properly bumped to subslot-aware versions.
""

(The unmasking of the library then triggers a "migration" to happy
portage updates that don't need human supervision)

In this case the library maintainers would have to touch random ebuilds,
I don't see any reason why maintainers of the affected packages would
disagree with such improvements.

As a criticism people might claim that it'll slow down new versions
appearing, but it's a one-time hit that'll make everything easier the
next bump, so I'd say don't worry about it.

wkr,

Patrick

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 688 bytes --]

Dnia 2013-09-26, o godz. 15:15:38
Patrick Lauer <patrick@gentoo.org> napisał(a):

> Thus I suggest declaring a policy:
> 
> ""
> Any library bump that would trigger revdep-rebuild should be done with
> the affected library package.mask'ed until all its consumers have been
> properly bumped to subslot-aware versions.
> ""
> 
> (The unmasking of the library then triggers a "migration" to happy
> portage updates that don't need human supervision)

How do we handle packages which install multiple libraries? I'm afraid
forcing such a policy and/or hurrying developers to adapt will only
cause more of poppler-like issues to occur.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

On Thu, 26 Sep 2013 09:53:22 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> How do we handle packages which install multiple libraries? I'm afraid
> forcing such a policy and/or hurrying developers to adapt will only
> cause more of poppler-like issues to occur.

You change subslot if at least one of the libraries breaks. A little
extra rebuilding doesn't hurt anyone, but breakages do.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Michael Palimaka]

On 26/09/2013 17:53, Michał Górny wrote:
> Dnia 2013-09-26, o godz. 15:15:38
> Patrick Lauer <patrick@gentoo.org> napisał(a):
>
>> Thus I suggest declaring a policy:
>>
>> ""
>> Any library bump that would trigger revdep-rebuild should be done with
>> the affected library package.mask'ed until all its consumers have been
>> properly bumped to subslot-aware versions.
>> ""
>>
>> (The unmasking of the library then triggers a "migration" to happy
>> portage updates that don't need human supervision)
>
> How do we handle packages which install multiple libraries? I'm afraid
> forcing such a policy and/or hurrying developers to adapt will only
> cause more of poppler-like issues to occur.
>
There isn't a 100% perfect solution currently, and I agree that hurrying 
people will simply move us from "not enough rebuilds" to "too many 
rebuilds".

Poppler was a great example of what can go wrong. Apart from people 
being forced to rebuild packages that link only against one of the 
stable interfaces, I even saw rebuilds forced for packages that didn't 
even link against the libraries.

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 306 bytes --]

On Thu, 26 Sep 2013 20:51:26 +1000
Michael Palimaka <kensington@gentoo.org> wrote:
> There isn't a 100% perfect solution currently, and I agree that
> hurrying people will simply move us from "not enough rebuilds" to
> "too many rebuilds".

This is still a huge improvement.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Michael Palimaka]

On 26/09/2013 20:55, Ciaran McCreesh wrote:
> On Thu, 26 Sep 2013 20:51:26 +1000
> Michael Palimaka <kensington@gentoo.org> wrote:
>> There isn't a 100% perfect solution currently, and I agree that
>> hurrying people will simply move us from "not enough rebuilds" to
>> "too many rebuilds".
>
> This is still a huge improvement.
>
At the same time, it's also a huge regression.

We relied on revdep-rebuild for a long time, and preserve-libs is 
enabled now by default. Widespread subslot usage will go a long way to 
improve the user experience, but adding them carelessly now will only 
mean a lot of extra work later.

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]

On Thu, 26 Sep 2013 21:14:11 +1000
Michael Palimaka <kensington@gentoo.org> wrote:
> On 26/09/2013 20:55, Ciaran McCreesh wrote:
> > On Thu, 26 Sep 2013 20:51:26 +1000
> > Michael Palimaka <kensington@gentoo.org> wrote:
> >> There isn't a 100% perfect solution currently, and I agree that
> >> hurrying people will simply move us from "not enough rebuilds" to
> >> "too many rebuilds".
> >
> > This is still a huge improvement.
>
> At the same time, it's also a huge regression.

It's only a regression if you think "occasional unnecessary recompiles"
is a problem. And if you do think it's a problem, there are better
things to change to "fix" it, such as allowing full recompiles to be
avoided for minor changes to a package.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

On 26 September 2013 19:53, Michał Górny <mgorny@gentoo.org> wrote:

> How do we handle packages which install multiple libraries? I'm afraid
> forcing such a policy and/or hurrying developers to adapt will only
> cause more of poppler-like issues to occur.
>

Can you give a an example package which:

- installs multiple libraries
- has an ABI that may change for only one of those libraries
- it is sane / plausible to expect one downstream dependent *not* to
forcibly rebuild as a result of a chane in one of those libaries
- it is sane / plausible to expect a different downstream to forcibly
rebuild as a result of changes in one of those libraries

I'm not saying they don't exist, but having reference cases would help
discussion.

I just don't see Gentoo really supporting such a scheme in the near future,
its rather complicated, you'd possibly need to have a "provides" map for
each and every ebuild*, and dependents would need to declare which specific
providers they're bound to, which would greatly complicate ebuild stuff.

And that provides map would eventually approximate a list of all files the
ebuild emits with version suffixes, which could be rather overwhelming.

This would be similar to how dependencies work on CPAN modules, where
modules are unaware of the tar.gz's their dependencies are shipped in, and
only declare dependencies on the modules in the tar.gz's, and then the
indexer maps those to the relevant packages ( which makes handling that a
bag of ferrets in gentoo without non-standard tools )


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 2290 bytes --]

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 26/09/13 06:51 AM, Michael Palimaka wrote:
> On 26/09/2013 17:53, Michał Górny wrote:
>> How do we handle packages which install multiple libraries? I'm
>> afraid forcing such a policy and/or hurrying developers to adapt
>> will only cause more of poppler-like issues to occur.
>> 
> There isn't a 100% perfect solution currently, and I agree that
> hurrying people will simply move us from "not enough rebuilds" to
> "too many rebuilds".

Enforcing consistency is much more important imo than "emerge -uDN
@world" efficiency.  For those users that need more efficiency they
can always get it by upgrading individual packages with
'--rebuild-ignore' or '--ignore-built-slot-operator-deps y' after
seeing what all is going to be rebuilt via 'emerge -uDNav'

To be honest, the main issue that I see keeping slot-operators from
being properly used has to do with virtuals, because subslots (or full
slots for that matter) do not propagate through them.  But that
doesn't hold back converting libs and rdeps for everything not falling
to a virtual.

> Poppler was a great example of what can go wrong. Apart from
> people being forced to rebuild packages that link only against one
> of the stable interfaces, I even saw rebuilds forced for packages
> that didn't even link against the libraries.

The latter in that case was a mis-use of the ':=' on the poppler atom
in *DEPEND.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)

iF4EAREIAAYFAlJEQOkACgkQ2ugaI38ACPCp1QEAsSX+efIdTGRZ94EIgyYzQcSF
4TeEmFvzanp5A/6DL94BAIVnw3ayjTmOmUYevRl/Hr0cEyVv4X9T+bFnhngW6Ops
=CsnH
-----END PGP SIGNATURE-----

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Michael Palimaka]

On 27/09/2013 00:12, Ian Stakenvicius wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 26/09/13 06:51 AM, Michael Palimaka wrote:
>> On 26/09/2013 17:53, Michał Górny wrote:
>>> How do we handle packages which install multiple libraries? I'm
>>> afraid forcing such a policy and/or hurrying developers to adapt
>>> will only cause more of poppler-like issues to occur.
>>>
>> There isn't a 100% perfect solution currently, and I agree that
>> hurrying people will simply move us from "not enough rebuilds" to
>> "too many rebuilds".
>
> Enforcing consistency is much more important imo than "emerge -uDN
> @world" efficiency.  For those users that need more efficiency they
> can always get it by upgrading individual packages with
> '--rebuild-ignore' or '--ignore-built-slot-operator-deps y' after
> seeing what all is going to be rebuilt via 'emerge -uDNav'
Why do you think striving for correct subslot usage will reduce 
consistency? I am not saying we should avoid subslots completely for a 
package because of some edge case, but excessive unnecessary rebuilds 
will push users towards disabling the feature.

Subslots for poppler was an improvement on the previous situation and a 
great idea for libpoppler consumers. For packages that that use one of 
the stable interfaces, rebuilding them needlessly is a major annoyance, 
and definitely not what subslots were intended for. Why can't we limit 
our subslot usage to where it's actually useful (consumers of the 
unstable library)? It doesn't have to be all-or-none.

What about when the subslot of boost was equal to ${PV}? Was it really a 
good idea to make everyone rebuild half their system for a bugfix 
release, without even checking if the ABI changed?

>> Poppler was a great example of what can go wrong. Apart from
>> people being forced to rebuild packages that link only against one
>> of the stable interfaces, I even saw rebuilds forced for packages
>> that didn't even link against the libraries.
>
> The latter in that case was a mis-use of the ':=' on the poppler atom
> in *DEPEND.
Yes, misuse is what I'm complaining about. When used correctly, subslots 
are great.

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Andreas K. Huettel]

Am Donnerstag, 26. September 2013, 16:04:22 schrieb Kent Fredric:
> On 26 September 2013 19:53, Michał Górny <mgorny@gentoo.org> wrote:
> > How do we handle packages which install multiple libraries? I'm afraid
> > forcing such a policy and/or hurrying developers to adapt will only
> > cause more of poppler-like issues to occur.
> 
> Can you give a an example package which:
> 
> - installs multiple libraries
> - has an ABI that may change for only one of those libraries
> - it is sane / plausible to expect one downstream dependent *not* to
> forcibly rebuild as a result of a chane in one of those libaries
> - it is sane / plausible to expect a different downstream to forcibly
> rebuild as a result of changes in one of those libraries
> 
> I'm not saying they don't exist, but having reference cases would help
> discussion.

app-text/poppler
https://bugs.gentoo.org/show_bug.cgi?id=462020

-- 
Andreas K. Huettel
Gentoo Linux developer (council, kde)
dilfridge@gentoo.org
http://www.akhuettel.de/

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Davide Pesavento]

On Thu, Sep 26, 2013 at 4:04 PM, Kent Fredric <kentfredric@gmail.com> wrote:
>
> On 26 September 2013 19:53, Michał Górny <mgorny@gentoo.org> wrote:
>>
>> How do we handle packages which install multiple libraries? I'm afraid
>> forcing such a policy and/or hurrying developers to adapt will only
>> cause more of poppler-like issues to occur.
>
>
> Can you give a an example package which:
>
> - installs multiple libraries
> - has an ABI that may change for only one of those libraries
> - it is sane / plausible to expect one downstream dependent *not* to
> forcibly rebuild as a result of a chane in one of those libaries
> - it is sane / plausible to expect a different downstream to forcibly
> rebuild as a result of changes in one of those libraries
>

dev-python/PyQt4

Each module is a separate library, and each has its own ABI that can
change independently from the others. Downstream projects that rely
only on PyQt4's python API are not affected by ABI changes, but those
(very few) that link against one or more modules (e.g. kde-base/pykde4
I think) must be rebuilt.

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 26/09/13 11:24 AM, Davide Pesavento wrote:
> On Thu, Sep 26, 2013 at 4:04 PM, Kent Fredric
> <kentfredric@gmail.com> wrote:
>> 
>> On 26 September 2013 19:53, Michał Górny <mgorny@gentoo.org>
>> wrote:
>>> 
>>> How do we handle packages which install multiple libraries? I'm
>>> afraid forcing such a policy and/or hurrying developers to
>>> adapt will only cause more of poppler-like issues to occur.
>> 
>> 
>> Can you give a an example package which:
>> 
>> - installs multiple libraries - has an ABI that may change for
>> only one of those libraries - it is sane / plausible to expect
>> one downstream dependent *not* to forcibly rebuild as a result of
>> a chane in one of those libaries - it is sane / plausible to
>> expect a different downstream to forcibly rebuild as a result of
>> changes in one of those libraries
>> 
> 
> dev-python/PyQt4
> 
> Each module is a separate library, and each has its own ABI that
> can change independently from the others. Downstream projects that
> rely only on PyQt4's python API are not affected by ABI changes,
> but those (very few) that link against one or more modules (e.g.
> kde-base/pykde4 I think) must be rebuilt.
> 

To better round off this example, I assume that the API itself also
changes as a whole, and slot-operators are the ideal means to force
rebuilds on all rdeps when that occurrs?

Otherwise, I don't see why this example couldn't be satisfied by
bumping subslot whenever any sub-ABI changes and only using
slot-operators in *DEPEND atoms on those very few packages that link
to the modules.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)

iF4EAREIAAYFAlJEcUAACgkQ2ugaI38ACPBgagD/QCVt4wgbzqykUf2aAi6iyFaR
ZQoI8NdXLuc2FvWZyqYBALGEOsrIGN+LnDndXhSaN1SId5P63ltU0ZQvByjiHqXN
=kl2D
-----END PGP SIGNATURE-----

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

On Thu, 26 Sep 2013 14:53:53 -0300
Alexis Ballier <aballier@gentoo.org> wrote:
> please spend your time on something useful: 
> fix bug #449094 and bug #462138 or propose something you think better
> for fixing the problems those bugs describe.

Those are both an awful lot of work for a very minimal gain for a very
small number of packages.

> subslots are far from perfect atm

But they're much better than not having them, and they're probably as
good as you're going to get any time soon.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Alexis Ballier]

please, not this again...

please spend your time on something useful: 
fix bug #449094 and bug #462138 or propose something you think better
for fixing the problems those bugs describe.

subslots are far from perfect atm

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 191 bytes --]

On Fri, 27 Sep 2013 06:00:01 +1200
Kent Fredric <kentfredric@gmail.com> wrote:
> ( virtual/perl-* is a maintenance nightmare )

virtual/perl-* is self-inflicted.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

On 27 September 2013 03:10, Andreas K. Huettel <dilfridge@gentoo.org> wrote:

>
> app-text/poppler
> https://bugs.gentoo.org/show_bug.cgi?id=462020


Reading this bug re-fuels my wish for virtuals to un-happen. Maintaining a
virtual with a massive list of || (  )  conditions is so so painful,
especially considering their nature being they're communicating information
thats better communicated in the inverse.

ie: every time there is a new iteration of Foo, one must iterate all the
virtuals that relate to it, and vet their need to be updated, and every
single update you do is just something thats easy to get wrong, when it
always seems to make so much more sense to say "X provides Y", especially
in the case where X provides a very large list of Y ( virtual/perl-* is a
maintenance nightmare )





-- 
Kent
<http://kent-fredric.fox.geek.nz>

[-- Attachment #2: Type: text/html, Size: 1466 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Alexis Ballier]

On Thu, 26 Sep 2013 18:53:14 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Thu, 26 Sep 2013 14:53:53 -0300
> Alexis Ballier <aballier@gentoo.org> wrote:
> > please spend your time on something useful: 
> > fix bug #449094 and bug #462138 or propose something you think
> > better for fixing the problems those bugs describe.
> 
> Those are both an awful lot of work for a very minimal gain for a very
> small number of packages.

those more or less are what is needed to have subslots working nicely
for the (counter)examples in this thread


> > subslots are far from perfect atm
> 
> But they're much better than not having them, and they're probably as
> good as you're going to get any time soon.

if you waste your time trolling like in the other thread some
weeks/months ago then yes nothing else but trolling will happen.

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1668 bytes --]

On Thu, 26 Sep 2013 15:06:44 -0300
Alexis Ballier <aballier@gentoo.org> wrote:
> On Thu, 26 Sep 2013 18:53:14 +0100
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > On Thu, 26 Sep 2013 14:53:53 -0300
> > Alexis Ballier <aballier@gentoo.org> wrote:
> > > please spend your time on something useful: 
> > > fix bug #449094 and bug #462138 or propose something you think
> > > better for fixing the problems those bugs describe.
> > 
> > Those are both an awful lot of work for a very minimal gain for a
> > very small number of packages.
> 
> those more or less are what is needed to have subslots working nicely
> for the (counter)examples in this thread

Are you sure? I've not seen a prototype or reference implementation of
either idea, nor a demonstration of how such features would work in
practice. We know from past experience that there is a big gap between
"I have an idea" to "this is a viable solution", and I've not seen
anyone do the work to move either bug from the former to the latter.

It's all very well claiming "this will fix everything!", but for
non-trivial features such as these, you really need to get at least a
rough implementation to try it out.

> > > subslots are far from perfect atm
> > 
> > But they're much better than not having them, and they're probably
> > as good as you're going to get any time soon.
> 
> if you waste your time trolling like in the other thread some
> weeks/months ago then yes nothing else but trolling will happen.

Accusations of trolling have nothing to do with getting a feature
implemented. Writing a prototype implementation does.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

On 27 September 2013 06:00, Kent Fredric <kentfredric@gmail.com> wrote:

>
> ie: every time there is a new iteration of Foo, one must iterate all the
> virtuals that relate to it, and vet their need to be updated, and every
> single update you do is just something thats easy to get wrong, when it
> always seems to make so much more sense to say "X provides Y", especially
> in the case where X provides a very large list of Y ( virtual/perl-* is a
> maintenance nightmare )
>

Also, can I make a suggestion that if we were to bring back PROVIDES=, that
PROVIDES *NOT* be permitted to provide arbitrary $CAT/$P ?

My very rusty memory of when I last saw that saw that as one sticking point.

It seems to me that PROVIDES would be more useful in a different hierarchy,
ie:
PROVIDES="cpan-module/CLASS::MOP/1.6" or something.


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1409 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

On 27 September 2013 05:57, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com>wrote:

> virtual/perl-* is self-inflicted.


How would you recommend it?

Like for instance, we really do need virtuals ( or equivalent ) for many
things in there, because those things may stop being part of dev-lang/perl
at a future time. One such example is Module-Build, which means any
packages that require Module-Build and don't declare virtual/Module-Build
in their dependency will be broken somewhere after perl 5.20




-- 
Kent

[-- Attachment #2: Type: text/html, Size: 961 bytes --]

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Kent Fredric <kentfredric@gmail.com> wrote:
>
> On 27 September 2013 05:57, Ciaran McCreesh
><ciaran.mccreesh@googlemail.com>wrote:
>
>> virtual/perl-* is self-inflicted.
>
> How would you recommend it?

For those which are provided by perl itself, you could have
a corresponding useflag of dev-lang/perl and make a use dependency:
If the main perl tarball does not provide the package, the perl ebuild
can pull in the corresponding package as a dependency.

IMHO, virtual/perl-* are only useful when there are cases that
a package must depend on a _particular version_ of that virtual:
I considered it always strange that most of the perl packages are
installed in duplicate only because the virtual version and the
version provided by perl do not match. This is appropriate if
something really needs a newer version, but not just because something
*must* depend on the virtual only because some perl versions do
not provide it.

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Arfrever Frehtes Taifersar Arahesis]

[-- Attachment #1: Type: Text/Plain, Size: 685 bytes --]

2013-09-26 17:04 Michael Palimaka napisał(a):
> What about when the subslot of boost was equal to ${PV}? Was it really a 
> good idea to make everyone rebuild half their system for a bugfix 
> release, without even checking if the ABI changed?

It is wrong example due to 2 reasons:
1. Subslot of dev-libs/boost::gentoo was never equal to ${PV}.
2. Sonames of Boost libraries actually include full ${PV}, so rebuilding of reverse dependencies
   would be necessary after upgrade from Boost 1.54.0 to hypothetical 1.54.1.

Although upstream rarely releases X.Y.Z with Z != 0 (last was 1.46.1).
http://www.boost.org/users/history/

--
Arfrever Frehtes Taifersar Arahesis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2518 bytes --]

On 27 September 2013 08:02, Martin Vaeth
<vaeth@mathematik.uni-wuerzburg.de>wrote:

> For those which are provided by perl itself, you could have
> a corresponding useflag of dev-lang/perl and make a use dependency:
> If the main perl tarball does not provide the package, the perl ebuild
> can pull in the corresponding package as a dependency.
>

That would be horrible. You'd have a massive list of USE flags, and you'd
want *all* of them by default, and they'd have to pull the deps as
PDEPENDS, because its impossible to have them as DEPENDS.

And that would mean: X package wants Module::Build, so not only do you have
to declare a dependency on Module::Build somehow, thats controlled by a USE
flag, so maybe rebuild the entirety of Perl, just to install one thing that
can be installed seperately from Perl.

Also, instead of having the logical thing when Module::Build does get fully
removed from perl, that we can simply remap the virtual to always pull from
perl-core/Module-Build, we'd have to re-write every package in tree that
used Module-Build.

I was asking for solutions that reduce work, not increase it =p

And when you wanted a specific version of a "dual life" module, you'd be
completely out of luck. ( This is a problem with Module::Build, Test-Simple
and ExtUtils::MakeMaker, because people regularly depend on specific
versions of those things )


> This is appropriate if
> something really needs a newer version, but not just because something
> *must* depend on the virtual only because some perl versions do
> not provide it.

Worse than that I'm afraid, things that are virtualled are virtualled
because upstream can and will depend on a specific version of that thing,
and time to time, those versions are available only via CPAN, not via the
present version of Perl

And other times, upstream will depend on an explicit version which is
available only in specific versions of perl, and virtuals are the only tool
we have at present to mitigate these problems.

And more, there is the growing list of modules that may be presently
installed with perl, but are slated to stop being shipped with perl in a
future release of perl. That means things that don't depend on the virtuals
*now* will be broken when that version of perl ships, and its much, much
easier to use the virtuals to resolve this problem, as opposed to tracking
down all the modules that need to be fixed, and inlining a big list of
conditional dependencies in each and every module that uses such a package.


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 3273 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 506 bytes --]

Am Donnerstag, 26. September 2013, 20:11:04 schrieb Kent Fredric:
> On 27 September 2013 05:57, Ciaran McCreesh
> 
> <ciaran.mccreesh@googlemail.com>wrote:
> > virtual/perl-* is self-inflicted.
> 
> How would you recommend it?
> 

How about doing someting eclass-based, similar to add_kdebase_dep, and let the 
eclass carry the list of modules present inside the base perl distro?

-- 
Andreas K. Huettel
Gentoo Linux developer (council, kde)
dilfridge@gentoo.org
http://www.akhuettel.de/

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Andreas K. Huettel]

[-- Attachment #1: Type: Text/Plain, Size: 934 bytes --]


> How do we handle packages which install multiple libraries? I'm afraid
> forcing such a policy and/or hurrying developers to adapt will only
> cause more of poppler-like issues to occur.

Well, the best technical solution for this is probably "subslot dictionaries" 
as suggested in bug 462138. Then, for each library a separate subslot could be 
declared, and programs can only be rebuilt if really needed. However, 

1) I see this as a measure only for special cases, since it once again 
increases the maintainer burden. Going from subslot to subslot dictionary 
should only be done in a package if it's really worth the effort. 

2) Maybe we should consider decoupling the subslot from the slot then. I.e. 
introducing a separate variable SUBSLOT.

https://bugs.gentoo.org/show_bug.cgi?id=462138

-- 
Andreas K. Huettel
Gentoo Linux developer (council, kde)
dilfridge@gentoo.org
http://www.akhuettel.de/

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]

Dnia 2013-09-26, o godz. 17:24:49
Davide Pesavento <pesa@gentoo.org> napisał(a):

> On Thu, Sep 26, 2013 at 4:04 PM, Kent Fredric <kentfredric@gmail.com> wrote:
> >
> > On 26 September 2013 19:53, Michał Górny <mgorny@gentoo.org> wrote:
> >>
> >> How do we handle packages which install multiple libraries? I'm afraid
> >> forcing such a policy and/or hurrying developers to adapt will only
> >> cause more of poppler-like issues to occur.
> >
> >
> > Can you give a an example package which:
> >
> > - installs multiple libraries
> > - has an ABI that may change for only one of those libraries
> > - it is sane / plausible to expect one downstream dependent *not* to
> > forcibly rebuild as a result of a chane in one of those libaries
> > - it is sane / plausible to expect a different downstream to forcibly
> > rebuild as a result of changes in one of those libraries
> >
> 
> dev-python/PyQt4
> 
> Each module is a separate library, and each has its own ABI that can
> change independently from the others. Downstream projects that rely
> only on PyQt4's python API are not affected by ABI changes, but those
> (very few) that link against one or more modules (e.g. kde-base/pykde4
> I think) must be rebuilt.

How often does ABI of pyqt4 libraries change in such a way that rebuild
of pykde4 is not required?

Looking at the dep:

>=dev-python/PyQt4-4.9.5[${PYTHON_USEDEP},dbus,declarative,script(+),sql,svg,webkit,X]

I'd think it's fairly rare when only the libraries not listed above
change ABI without any of the remaining ones changing it.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Sorry if this is duplicate: I repost since I cannot see it after
a few hours.

Kent Fredric <kentfredric@gmail.com> wrote:
>
> On 27 September 2013 08:02, Martin Vaeth
><vaeth@mathematik.uni-wuerzburg.de>wrote:
>
>> For those which are provided by perl itself, you could have
>> a corresponding useflag of dev-lang/perl and make a use dependency:
>> If the main perl tarball does not provide the package, the perl ebuild
>> can pull in the corresponding package as a dependency.
>
> That would be horrible. You'd have a massive list of USE flags, and you'd
> want *all* of them by default, and they'd have to pull the deps as
> PDEPENDS, because its impossible to have them as DEPENDS.

Why is this horrible?  Perhaps there is a misunderstanding.
You would need only those useflags which are possibly not provided
by the corresponding perl version: The package dependent on a feature
could just use e.g. DEPEND=dev-lang/perl[perl_module_Term_ANSIColor(+)]
And concerning the number of use-flags, it might be worth thinking
about a USE_EXPAND="perl_module".

> so maybe rebuild the entirety of Perl, just to install one thing that
> can be installed seperately from Perl.

If the common flags are enabled by default, this would only involve
users who have a serious reason (e.g. space issues) to disable some
modules.  Yes, these users would have to rebuild unnecessarily if
they change their mind later on.  I would compare disabling these
modules with something like setting USE=minimal for other packages;
users should know what they are doing if they use this.

> Also, instead of having the logical thing when Module::Build does get fully
> removed from perl, that we can simply remap the virtual to always pull from
> perl-core/Module-Build, we'd have to re-write every package in tree that
> used Module-Build.

My idea is to keep this in the perl ebuilds "forever":
According to the perlversions which are still in the tree, it will take
dozens of years until all perl versions providing Module-Build have vanished
from the  tree, and only afterwards it makes sense to think about
such a change.

> This is a problem with Module::Build, Test-Simple
> and ExtUtils::MakeMaker, because people regularly depend on specific
> versions of those things

My suggestion was explicitly about modules for which ebuilds do
not require an explicit version. For those few(!) modules for which
particular versions are needed, perhaps the virtual might be kept.
Alternatively, for such cases it might make sense to depend
directly on the package, since the probability to save duplicate
installation in such a case is rather low, anyway.

> Worse than that I'm afraid, things that are virtualled are virtualled
> because upstream can and will depend on a specific version of that thing

As mentioned above, this involves only a relatively small number of
virtuals. Here is how I got the list:

eix --print-all-depends |  sed 's/"//g' \
  grep -o '[^ ]*virtual/perl-[^ ]*-[0-9][^ ]*' |sort -u

And I guess that in this list of (for the main tree, but even with
eix -Z ... the list is hardly longer) 62 packages
actually many fall into the class where the minimal version is
provided by current stable perl version anyway so that actually the
minimal version dependency is redundant.

> And more, there is the growing list of modules that may be presently
> installed with perl, but are slated to stop being shipped with perl
> [...] and inlining a big list of
> conditional dependencies in each and every module that uses such a package.

I was not suggesting inlining the list into the dependency but
only inlining the USE-flag into the (single) perl ebuild.
Currently if I have a package which needs e.g. Term-ANSI-Color,
but not in a particular version, if I do not want to install an
unnecessary duplicate version, I must inline a dependency like
|| ( >=dev-lang/perl-5.14 virtual/perl-Term-ANSIColor )
and possibly change this if perl-5.20 does no longer contain
perl-Term-ANSIColor.

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Matt Turner]

On Fri, Sep 27, 2013 at 12:48 PM, Martin Vaeth
<vaeth@mathematik.uni-wuerzburg.de> wrote:
> I was not suggesting inlining the list into the dependency but
> only inlining the USE-flag into the (single) perl ebuild.
> Currently if I have a package which needs e.g. Term-ANSI-Color,
> but not in a particular version, if I do not want to install an
> unnecessary duplicate version, I must inline a dependency like
> || ( >=dev-lang/perl-5.14 virtual/perl-Term-ANSIColor )
> and possibly change this if perl-5.20 does no longer contain
> perl-Term-ANSIColor.

Isn't that exactly what the virtual should do?

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 8262 bytes --]

On 28 September 2013 07:48, Martin Vaeth
<vaeth@mathematik.uni-wuerzburg.de>wrote:

> Sorry if this is duplicate: I repost since I cannot see it after
> a few hours.
>
> Kent Fredric <kentfredric@gmail.com> wrote:
> >
> > On 27 September 2013 08:02, Martin Vaeth
> ><vaeth@mathematik.uni-wuerzburg.de>wrote:
> >
> >> For those which are provided by perl itself, you could have
> >> a corresponding useflag of dev-lang/perl and make a use dependency:
> >> If the main perl tarball does not provide the package, the perl ebuild
> >> can pull in the corresponding package as a dependency.
> >
> > That would be horrible. You'd have a massive list of USE flags, and you'd
> > want *all* of them by default, and they'd have to pull the deps as
> > PDEPENDS, because its impossible to have them as DEPENDS.
>
> Why is this horrible?  Perhaps there is a misunderstanding.
> You would need only those useflags which are possibly not provided
> by the corresponding perl version: The package dependent on a feature
> could just use e.g. DEPEND=dev-lang/perl[perl_module_Term_ANSIColor(+)]
> And concerning the number of use-flags, it might be worth thinking
> about a USE_EXPAND="perl_module".
>
>
Its just a "Perl module" here can be only "1 file",  and worse, you'll have
to mentally make sure the USE flags have the right interdependencies for
USE flags that require other USE flags to work.

And rebuilding all of perl to change the presence of one file, gives us a
lot of cost, and no benefit. And it still has the problem if a user wants a
version of something that we don't know where its going to come from for
all time, ie: you'll still need logic like "|| (
dev-lang/perl[perl_module_Term_ANSIColor(-)]   perl-core/Term-ANSIColor )"
to just deal with the reality of what upstream are asking for.

This solution is presently worse than virtuals, because it requires every
ebuild to be aware of the mess of how resolving "dual life" dependencies
work.

> so maybe rebuild the entirety of Perl, just to install one thing that
> > can be installed seperately from Perl.
>
> If the common flags are enabled by default, this would only involve
> users who have a serious reason (e.g. space issues) to disable some
> modules.  Yes, these users would have to rebuild unnecessarily if
> they change their mind later on.  I would compare disabling these
> modules with something like setting USE=minimal for other packages;
> users should know what they are doing if they use this.
>

The problem there is you're breaking upstream, because you're breaking how
upstream perceives perl to work.

If such a list of USE flags existed, it would be a very strong
recommendation that they *ALL* be turned on.  And again, this essentially
says "modules that are dual lifed are pretended not to be so", unless you
combine this approach with PDEPEND

And pulling in perl-core/Whatever by doing
=dev-lang/perl[perl_module_whatever] is just a nastier form of
virtual/perl-Whatever, with the limitation that you're completely
destroying any version support.

> Also, instead of having the logical thing when Module::Build does get
> fully
> > removed from perl, that we can simply remap the virtual to always pull
> from
> > perl-core/Module-Build, we'd have to re-write every package in tree that
> > used Module-Build.
>
> My idea is to keep this in the perl ebuilds "forever":
> According to the perlversions which are still in the tree, it will take
> dozens of years until all perl versions providing Module-Build have
> vanished
> from the  tree, and only afterwards it makes sense to think about
> such a change.
>
> > This is a problem with Module::Build, Test-Simple
> > and ExtUtils::MakeMaker, because people regularly depend on specific
> > versions of those things
>
> My suggestion was explicitly about modules for which ebuilds do
> not require an explicit version. For those few(!) modules for which
> particular versions are needed, perhaps the virtual might be kept.
> Alternatively, for such cases it might make sense to depend
> directly on the package, since the probability to save duplicate
> installation in such a case is rather low, anyway


Thats not really the issue, the issue is that because the modules *ARE *deemed
dual life by upstream, that is, it is expected that end users can depend on
a specific version of a module that exists in both perl itself, and as a
standalone, that end users *may* depend on such things and expect that to
work.

In the event new modules do hit tree need something that is dual lifed, it
is much less effort to add a virtual to the tree to communicate the
dependency properly, much less effort than rehashing dev-lang/perl itself,
adding a new USE flag, and causing a domino of recompiles to deploy a
hand-full of files that may not even need compiling to work.


> > Worse than that I'm afraid, things that are virtualled are virtualled
> > because upstream can and will depend on a specific version of that thing
>
> As mentioned above, this involves only a relatively small number of
> virtuals. Here is how I got the list:
>
> eix --print-all-depends |  sed 's/"//g' \
>   grep -o '[^ ]*virtual/perl-[^ ]*-[0-9][^ ]*' |sort -u
>
>
I get 566 for me. Thats a bit longer =)  . And note, you're showing the
dependencies, not the dependants.

If you remove the unique criteria, you get a lovely 20260 lines of output!

And I guess that in this list of (for the main tree, but even with
> eix -Z ... the list is hardly longer) 62 packages
> actually many fall into the class where the minimal version is
> provided by current stable perl version anyway so that actually the
> minimal version dependency is redundant.
>
> > And more, there is the growing list of modules that may be presently
> > installed with perl, but are slated to stop being shipped with perl
> > [...] and inlining a big list of
> > conditional dependencies in each and every module that uses such a
> package.
>
> I was not suggesting inlining the list into the dependency but
> only inlining the USE-flag into the (single) perl ebuild.
> Currently if I have a package which needs e.g. Term-ANSI-Color,
> but not in a particular version, if I do not want to install an
> unnecessary duplicate version, I must inline a dependency like
> || ( >=dev-lang/perl-5.14 virtual/perl-Term-ANSIColor )
> and possibly change this if perl-5.20 does no longer contain
> perl-Term-ANSIColor.
>
>
>
|| ( >=dev-lang/perl-5.14 virtual/perl-Term-ANSIColor )

That is plain wrong imo. You're prematurely optimising the dependency.
There is no guarantee any future version of Perl will contain it. That is
*why* we have the virtual.

The virtuals job is to invoke dependency on dev-lang/perl as much as
possible, and default to perl-core/* when dev-lang/perl does not provide
the version matched on the virtual.

This may not be important for *your* dist, but it is very necessary for
other things. Especially in the case of blocking versions.

I mean, the reason virtuals suck is predominantly the fact we have to
update the conditionals to accurately reflect the version on the virtual,
so that virtuals pull in perl-core/* if the version in dev-lang/perl
doesn't match the version.

Literally, the logical replacement for that is inlining that conditional in
every .ebuild that needs it.

We could, at very best, step around the problem by having an eclass that
translates a special variable, say,
PERL_MODULE_COREDEPS="Term::ANSIColor@3.2"  or something, and that then
translates those dependencies into the relevant switches. And that is
something I really want to avoid, mostly, because the massive amount of
moving parts required to implement such a feature.

I'd sooner opt for a way to represent the ebuild as a template, and then
have a process that generates ebuilds from the ebuild templates,
automatically -r bumping things as it found dependency changes.

At least that way, I wouldn't be limited to nasty arcane bash tricks that
could explode on a user  with relative ease.

I'd also sooner consider attempting to eliminate the need for virtuals by
unilaterally depending on perl-core/* , and vivifying perl-core/ from
dev-lang/perl sources as needed. Just the maint overhead for that is
slightly high, and entirely unsupported by upstream.




-- 
Kent

[-- Attachment #2: Type: text/html, Size: 10579 bytes --]

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Matt Turner <mattst88@gentoo.org> wrote:
>
>> || ( >=dev-lang/perl-5.14 virtual/perl-Term-ANSIColor )
>> and possibly change this if perl-5.20 does no longer contain
>> perl-Term-ANSIColor.
>
> Isn't that exactly what the virtual should do?

One can discuss what it *should* do, but it is certainly not
what happens, because the virtual exists in different versions:

If you only depend on virtual/perl-Term-ANSIColor,
it would currently install perl-core/Term-ANSIColor-4.20
*in addition* to the one provided by perl-5.14/5.16
(which is Term-ANSIColor-4.02 or -3.0.0, respectively).

Note that it would be stupid to depend on e.g.
=virtual/perl-Term-ANSIColor-4.02
for several reason:

1. The virtual does not even exist :)
2. It would collide with ebuilds depending on other versions.
3. This version is only reasonable if perl-5.16 is the
perl version which the user has installed.

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Kent Fredric <kentfredric@gmail.com> wrote:
>
> you'll still need logic like "|| (
> dev-lang/perl[perl_module_Term_ANSIColor(-)]   perl-core/Term-ANSIColor )"
> to just deal with the reality of what upstream are asking for.

My point is that the perl ebuild need not necessarily follow upstream:
It follows what the perl *ebuild* provides.
If upstream decides to remove a package, you can just pull it
from the ebuild (with PDEPEND, I agree) nevertheless.

> If such a list of USE flags existed, it would be a very strong
> recommendation that they *ALL* be turned on.

Yes, this is why I said manually disabling is comparable with
setting USE=minimal: for most users not recommended unless you
really have the necessity to build a minimal system for some reason.
So I would not care too much about occassional unnecessary recompilation
of perl itself only for the small numbers of users having such a necessity.

> And pulling in perl-core/Whatever by doing
>=dev-lang/perl[perl_module_whatever] is just a nastier form of
> virtual/perl-Whatever, with the limitation that you're completely
> destroying any version support.

If you need version support you still can depend on perl-core or virtual/*
but currently there is no way to explicitly prefer the perl-provided version
in the dependency (unless you code it manually).

> Thats not really the issue, the issue is that because the modules *ARE *deemed
> dual life by upstream, that is, it is expected that end users can depend on
> a specific version of a module that exists in both perl itself, and as a
> standalone, that end users *may* depend on such things and expect that to
> work.

Yes, he may depend on the explicit perl-core/* with version
(and perhaps also some virtual/* where it is likely that such
an explicit version is provided by perl itself - probably only
very few):

>> As mentioned above, this involves only a relatively small number of
>> virtuals. Here is how I got the list:
>>
>> eix --print-all-depends |  sed 's/"//g' \
>>   grep -o '[^ ]*virtual/perl-[^ ]*-[0-9][^ ]*' |sort -u
>>
> I get 566 for me.

different *minimal versions* but much less (IIRC 62) packages
which is already now lower than the number of virtual/perl-*.
Moreover, many of the minimal versions are probably meanwhile
redundant (since the highest version is stable anyway and often
already provided by current stable perl).
Since most packages occur only once or twice, I guess that if the
redundancy is eliminated you end up with only 10-20 for which a
virtual might really make sense (or, alternatively, for which you
must directly depend on perl-core/*, since the probability to
just have the correct version in perl is rather low anyway).

> And note, you're showing the dependencies, not the dependants.

This is the point, because only this is what is interesting:
You do not need a virtual with version number if absolutely nothing
is using it.

> If you remove the unique criteria, you get a lovely 20260 lines
> of output!

This number has no meaning. Moreover, if you should decide
to change the way how modules depend, this is a question
of writing a single perl-script ;)  which changes the style
in all ebuilds. I can gladly provide such a script if you want.

>|| ( >=dev-lang/perl-5.14 virtual/perl-Term-ANSIColor )
>
> That is plain wrong imo. You're prematurely optimising the dependency.

The alternative is to pull in a duplicated installation which is
completely superfluous, since it is already installed by most
perl versions,

> There is no guarantee any future version of Perl will contain it.

That's why it is necessary to manually update the dependency -
a lot more work than to depend on dev-lang/perl[Term-ANSIColor]
and only edit a future perl ebuild to pull in the package.

> The virtuals job is to invoke dependency on dev-lang/perl as much as
> possible, and default to perl-core/* when dev-lang/perl does not provide
> the version matched on the virtual.

In this example (one of many) the version plays no role
for the dependency, but nevertheless the virtual/... implementation
will pull in an unnecessary package.

> I'd also sooner consider attempting to eliminate the need for virtuals by
> unilaterally depending on perl-core/* , and vivifying perl-core/ from
> dev-lang/perl sources as needed.

This breaks proper support for building/using binary packages for
perl-core/* since the installed files will depend on which packages
are installed at build-time.

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Kent Fredric <kentfredric@gmail.com> wrote:
>
> I mean, the reason virtuals suck is predominantly the fact we have to
> update the conditionals to accurately reflect the version on the virtual

Concerning the eclass idea which was already mentioned and
which is perhaps even better than my suggeestion from the
other posting, since it avoids some of the disadvantages:

> by having an eclass that translates a special variable, say,
> PERL_MODULE_COREDEPS="Term::ANSIColor@3.2"  or something

I would keep it simple like

RDEPEND="$(perl-dep Term-ANSIColor)
  $(perl-dep ">=Module-Load-0.240.0")"

which would then translate into something like

RDEPEND="( || ( ( >=perl-5.12 <=perl-5.18.1 ) perl-core/Term-ANSIColor )
  || ( >=perl-5.18 >=perl-core/Module-Load-0.240.0 )"

(or "|| ( ( perl-5.12* perl-5.14* ...
|| ( perl-5.18* ...)" -
subject to further discussion)

Of course, perl-dep might also loop over its arguments...
Transfer of current ebuilds into the new format would be trivial:
s{(\S*)virtual/perl-(\S*)}{\$\(perl-dep \"$1$2\"\)}g
(and adding the inherit if at least one match occured).

> something I really want to avoid, mostly, because the massive amount of
> moving parts required to implement such a feature.

If the eclass is written generically you only have to fill in
one list for each perl-version (further simplifications of the data
format are thinkable) containg the provided packages e.g. in
the form category/package-version
If there is interest, I might volunteer to provide a first form
of such an eclass (though I can make no promises in the moment).

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Davide Pesavento]

On Thu, Sep 26, 2013 at 7:39 PM, Ian Stakenvicius <axs@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 26/09/13 11:24 AM, Davide Pesavento wrote:
>> On Thu, Sep 26, 2013 at 4:04 PM, Kent Fredric
>> <kentfredric@gmail.com> wrote:
>>>
>>> On 26 September 2013 19:53, Michał Górny <mgorny@gentoo.org>
>>> wrote:
>>>>
>>>> How do we handle packages which install multiple libraries? I'm
>>>> afraid forcing such a policy and/or hurrying developers to
>>>> adapt will only cause more of poppler-like issues to occur.
>>>
>>>
>>> Can you give a an example package which:
>>>
>>> - installs multiple libraries - has an ABI that may change for
>>> only one of those libraries - it is sane / plausible to expect
>>> one downstream dependent *not* to forcibly rebuild as a result of
>>> a chane in one of those libaries - it is sane / plausible to
>>> expect a different downstream to forcibly rebuild as a result of
>>> changes in one of those libraries
>>>
>>
>> dev-python/PyQt4
>>
>> Each module is a separate library, and each has its own ABI that
>> can change independently from the others. Downstream projects that
>> rely only on PyQt4's python API are not affected by ABI changes,
>> but those (very few) that link against one or more modules (e.g.
>> kde-base/pykde4 I think) must be rebuilt.
>>
>
> To better round off this example, I assume that the API itself also
> changes as a whole, and slot-operators are the ideal means to force
> rebuilds on all rdeps when that occurrs?
>

Not sure what you mean by "the API also changes as a whole".

The ABI can change without the API changing, if that's what you're
asking, but I'm sure you already know that...

> Otherwise, I don't see why this example couldn't be satisfied by
> bumping subslot whenever any sub-ABI changes and only using
> slot-operators in *DEPEND atoms on those very few packages that link
> to the modules.
>

You'd force unnecessary rebuilds for reverse deps not using the
library that changed ABI, just like the poppler case.

And there's an additional "problem" here: from what I've seen, most
people blindly add subslot operators in *DEPEND when they see that the
dep has gained a subslot, even when it's not needed at all (e.g.
python-only packages in the PyQt4 case).

Thanks,
Davide

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Davide Pesavento]

On Fri, Sep 27, 2013 at 4:44 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Dnia 2013-09-26, o godz. 17:24:49
> Davide Pesavento <pesa@gentoo.org> napisał(a):
>
>> On Thu, Sep 26, 2013 at 4:04 PM, Kent Fredric <kentfredric@gmail.com> wrote:
>> >
>> > On 26 September 2013 19:53, Michał Górny <mgorny@gentoo.org> wrote:
>> >>
>> >> How do we handle packages which install multiple libraries? I'm afraid
>> >> forcing such a policy and/or hurrying developers to adapt will only
>> >> cause more of poppler-like issues to occur.
>> >
>> >
>> > Can you give a an example package which:
>> >
>> > - installs multiple libraries
>> > - has an ABI that may change for only one of those libraries
>> > - it is sane / plausible to expect one downstream dependent *not* to
>> > forcibly rebuild as a result of a chane in one of those libaries
>> > - it is sane / plausible to expect a different downstream to forcibly
>> > rebuild as a result of changes in one of those libraries
>> >
>>
>> dev-python/PyQt4
>>
>> Each module is a separate library, and each has its own ABI that can
>> change independently from the others. Downstream projects that rely
>> only on PyQt4's python API are not affected by ABI changes, but those
>> (very few) that link against one or more modules (e.g. kde-base/pykde4
>> I think) must be rebuilt.
>
> How often does ABI of pyqt4 libraries change in such a way that rebuild
> of pykde4 is not required?

Practically never (see below).

>
> Looking at the dep:
>
>>=dev-python/PyQt4-4.9.5[${PYTHON_USEDEP},dbus,declarative,script(+),sql,svg,webkit,X]
>
> I'd think it's fairly rare when only the libraries not listed above
> change ABI without any of the remaining ones changing it.
>

Actually, most PyQt4 libraries never changed their ABIs since the
initial 4.0 release, so yes, it's "fairly rare" :)
That's also one of the reasons why I never bothered to add a subslot to PyQt4.

Thanks,
Davide

Re: [gentoo-dev] [RFC] Policy for migrating library consumers to subslots

[Ian Stakenvicius]

On 2013-09-28, at 9:43 AM, Davide Pesavento <pesa@gentoo.org> wrote:

> On Fri, Sep 27, 2013 at 4:44 PM, Michał Górny <mgorny@gentoo.org> wrote:
>> Dnia 2013-09-26, o godz. 17:24:49
>> Davide Pesavento <pesa@gentoo.org> napisał(a):
>> 
>>> On Thu, Sep 26, 2013 at 4:04 PM, Kent Fredric <kentfredric@gmail.com> wrote:
>>>> 
>>>> On 26 September 2013 19:53, Michał Górny <mgorny@gentoo.org> wrote:
>>>>> 
>>>>> How do we handle packages which install multiple libraries? I'm afraid
>>>>> forcing such a policy and/or hurrying developers to adapt will only
>>>>> cause more of poppler-like issues to occur.
>>>> 
>>>> 
>>>> Can you give a an example package which:
>>>> 
>>>> - installs multiple libraries
>>>> - has an ABI that may change for only one of those libraries
>>>> - it is sane / plausible to expect one downstream dependent *not* to
>>>> forcibly rebuild as a result of a chane in one of those libaries
>>>> - it is sane / plausible to expect a different downstream to forcibly
>>>> rebuild as a result of changes in one of those libraries
>>> 
>>> dev-python/PyQt4
>>> 
>>> Each module is a separate library, and each has its own ABI that can
>>> change independently from the others. Downstream projects that rely
>>> only on PyQt4's python API are not affected by ABI changes, but those
>>> (very few) that link against one or more modules (e.g. kde-base/pykde4
>>> I think) must be rebuilt.
>> 
>> How often does ABI of pyqt4 libraries change in such a way that rebuild
>> of pykde4 is not required?
> 
> Practically never (see below).
> 
>> 
>> Looking at the dep:
>> 
>>> =dev-python/PyQt4-4.9.5[${PYTHON_USEDEP},dbus,declarative,script(+),sql,svg,webkit,X]
>> 
>> I'd think it's fairly rare when only the libraries not listed above
>> change ABI without any of the remaining ones changing it.
> 
> Actually, most PyQt4 libraries never changed their ABIs since the
> initial 4.0 release, so yes, it's "fairly rare" :)
> That's also one of the reasons why I never bothered to add a subslot to PyQt4.
> 
> Thanks,
> Davide
> 


it is very important to note that slot operators should be used even when the atom is unlikely to ever bump ABI or SONAME.  it is the rare big update that breaks everything -- think the expat update fiasco from a few years back.

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1849 bytes --]

On 28 September 2013 22:31, Martin Vaeth
<vaeth@mathematik.uni-wuerzburg.de>wrote:

>
> Note that it would be stupid to depend on e.g.
> =virtual/perl-Term-ANSIColor-4.02
> for several reason:
>
> 1. The virtual does not even exist :)
>

Nope, it does. Its just called =virtual/perl-Term-ANSIColor-4.20.0 ,
because upstreams versioning semantics are grossly different to gentoos, so
we translate upstreams floating point versions to multipart versions so
that version cross-dependency semantics stay the same.


> 2. It would collide with ebuilds depending on other versions.
>

Nope, it would only collide in the case other ebuilds depended on a
specific version, or blocked a specific version.

This is seen as a "feature", because of "foo" needs "<= y-nnn" you don't
want to have to logically tract that fact in every other package that needs
"y"


> 3. This version is only reasonable if perl-5.16 is the
> perl version which the user has installed


And when perl 5.18 hits tree, if you want exactly 4.20 or larger, you'll
need to change your dependency string in your package to incorporate this
fact, instead of saying ">=virtual/perl-Term-ANSIColor-4.20.0" you'll have
to do "|| ( perl-5.18 etc etc )" , which is pushing the dependency
management of all the virtuals into the pacakges that are requring them.

Virtuals are much more convenient here, because if we need to change the
provider of a single version, we change it in *one* place, instead of
having to modify every ebuild in tree that needed it.

( Actually, thats a bug still, because corelist -a says 4.20.0 should be
available in 5.18, but the virtual for 4.20.0 doesn't yet have perl-5.18
listed as a provider, though I'm glad it can be fixed once, there, instead
of having to fix it in every package that depended on 4.20.0 prior to the
arrival of 5.18 in tree )



-- 
Kent

[-- Attachment #2: Type: text/html, Size: 2824 bytes --]

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 8044 bytes --]

On 28 September 2013 22:46, Martin Vaeth
<vaeth@mathematik.uni-wuerzburg.de>wrote:

> Kent Fredric <kentfredric@gmail.com> wrote:
> >
> > you'll still need logic like "|| (
> > dev-lang/perl[perl_module_Term_ANSIColor(-)]   perl-core/Term-ANSIColor
> )"
> > to just deal with the reality of what upstream are asking for.
>
> My point is that the perl ebuild need not necessarily follow upstream:
> It follows what the perl *ebuild* provides.
> If upstream decides to remove a package, you can just pull it
> from the ebuild (with PDEPEND, I agree) nevertheless.
>
>
So are you basically suggesting that for dual life modules, we simply
ignore that they're dual-lifeable, and then when upstream splits a package
from core so that its no longer dual life, that we simply ignore that too,
and fake it still being core for the forseeable future?


> If such a list of USE flags existed, it would be a very strong
> > recommendation that they *ALL* be turned on.
>
> Yes, this is why I said manually disabling is comparable with
> setting USE=minimal: for most users not recommended unless you
> really have the necessity to build a minimal system for some reason.
> So I would not care too much about occassional unnecessary recompilation
> of perl itself only for the small numbers of users having such a necessity.
>
>
IMO, really, the uses would be forced enabled for all users, because they
should never be disabled. If they're part of the Perl built itself, they
should get installed. Period.

Then by forcing them on all the time, you can use them how you were
initially suggesting, as a way to track interdependencies between versions
of perl ( ie: When perl itself stops being able to provide something, the
USE flag goes away ... but thats messy as hell, and an abuse of the entire
purpose of USE flags, to control features, not to simply track properties
of a package for the purpose of cross dependencies )

For that, a slot-dict approach is far more sane.


> > And pulling in perl-core/Whatever by doing
> >=dev-lang/perl[perl_module_whatever] is just a nastier form of
> > virtual/perl-Whatever, with the limitation that you're completely
> > destroying any version support.
>
> If you need version support you still can depend on perl-core or virtual/*
> but currently there is no way to explicitly prefer the perl-provided
> version
> in the dependency (unless you code it manually).
>
>
But why would you "depend on the perl-provided version" , that mentality is
nowhere upstream, and nowhere downstream.

Are you saying that, if something is provided by perl core, that we must
never update to a cpan version?

You realise thats breaking how upstream thinks toolchains work right?

Because even CPAN and friends like that will upgrade things in core to
their cpan versions where possible.

There is *no* way for upstream to declare "I want whatever version of X is
in your current perl", they can only state "I want X" or fail to state they
want X, and assume toolchain does the right thing.  Even then, that will
result in tools using more recent versions of things from CPAN.

> Thats not really the issue, the issue is that because the modules *ARE
> *deemed
> > dual life by upstream, that is, it is expected that end users can depend
> on
> > a specific version of a module that exists in both perl itself, and as a
> > standalone, that end users *may* depend on such things and expect that to
> > work.
>
> Yes, he may depend on the explicit perl-core/* with version
> (and perhaps also some virtual/* where it is likely that such
> an explicit version is provided by perl itself - probably only
> very few):
>

But that raises a problem, because some versions that end users may depend
on are *NOT* available as perl-core/* , they are only available in a
specific incarnation of Perl.

Or sometimes, a version will appear on CPAN, and people will start
depending on it ( requiring you to invoke a perl-core/* dependency ), and
later, that version will be shipped in perl itself, resulting in the need
to retroactively modify every ebuild that depended on perl-core/* of those
versions to use the perl one instead.

Because otherwise, you'll have end users complaining they have to install
perl-core/Term-ANSIColor-4.20.0 when they're using 5.18.0, which comes with
that version anyway.


> > And note, you're showing the dependencies, not the dependants.
>
> This is the point, because only this is what is interesting:
> You do not need a virtual with version number if absolutely nothing
> is using it.
>
> > If you remove the unique criteria, you get a lovely 20260 lines
> > of output!
>
> This number has no meaning. Moreover, if you should decide
> to change the way how modules depend, this is a question
> of writing a single perl-script ;)  which changes the style
> in all ebuilds. I can gladly provide such a script if you want.
>
>
You're missing a very important point: Every single line of output without
the uniq constraint is a package depending on a virtual.

The virtual is managing the need to have a conditional dependency based on
the version of perl installed.

You would need to ,without virtuals, modify *EVERY* ebuild containg a
dependency on a virtual, to contain the respective conditional dependency
enshrined in the virtual.

Not only that, instead of what we currently do, modify the indivual
virtuals to adapt to new perl releases, with the conditional codified in
the .ebuild, you would need to modify *EVERY* ebuild that had such a
dependency after *EVERY* perl release.

Not sure about you, but the idea of modifying 20,000 lines of code instead
of 1, is something I don't look on fondly.

>|| ( >=dev-lang/perl-5.14 virtual/perl-Term-ANSIColor )
> >
> > That is plain wrong imo. You're prematurely optimising the dependency.
>
> The alternative is to pull in a duplicated installation which is
> completely superfluous, since it is already installed by most
> perl versions,
>

You're not though. Thats the point. Virtuals are doing exactly that.
They're just doing it in the virtual instead of your ebuild.

If you are installing virtual Y version X, if you are on a version of Perl
that contains Y version X, the virtual avoids the need to install the
respective perl-core/X.

Thats exactly how they work, and is exactly their point.

If you want proof of this:

eix --only-names -IcC virtual | grep 'virtual/perl-'  | wc -l
50

 eix --only-names -IcC perl-core | wc -l
35

So thats 15 virtuals that are still not pulling in any superflous
perl-core/*

They don't need to, they've short-circuited to using perl instead.

If you don't like that packages get updated, and pull newer-that-core
versions of things, there's nothing to stop you grepping virtual/perl-*,
and masking versions that don't match an "=your current version of perl".

And then it will cease to pull in *any* perl-core/*

Lots of things will not work if you do that, but you can do that.

Sure, the perl-core/ family will be around for a little longer, but they
should become candidates for depcleaning as soon as the conditionals
default to perl.

You could also mask perl-core/* and see what that does, though I'd wager
you'll see a lot of things breaking.

In this example (one of many) the version plays no role
> for the dependency, but nevertheless the virtual/... implementation
> will pull in an unnecessary package.
>
>
As stated above, if your objection is simply getting new versions of things
in core needlessly, be my guest, disable that mechanic. You can do so right
now. Though I'd advise against it, because that mechanic is nessecary for
many things, hence, why we have it.

> I'd also sooner consider attempting to eliminate the need for virtuals by
> > unilaterally depending on perl-core/* , and vivifying perl-core/ from
> > dev-lang/perl sources as needed.
>
> This breaks proper support for building/using binary packages for
> perl-core/* since the installed files will depend on which packages
> are installed at build-time.


I'm not really sure what you're sayng here. Sorry :/


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 10801 bytes --]

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

On Sun, 29 Sep 2013 08:47:52 +1300
Kent Fredric <kentfredric@gmail.com> wrote:
> For that, a slot-dict approach is far more sane.

The only reason anyone can make that claim is that no-one really knows
what slot dictionaries are or how they'd work in practice. Until
there's a rough description of how they work and a prototype
implementation, "subslot dictionaries" is like "magic beans".

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2192 bytes --]

On 28 September 2013 23:36, Martin Vaeth
<vaeth@mathematik.uni-wuerzburg.de>wrote:

>
> Concerning the eclass idea which was already mentioned and
> which is perhaps even better than my suggeestion from the
> other posting, since it avoids some of the disadvantages:
>
> > by having an eclass that translates a special variable, say,
> > PERL_MODULE_COREDEPS="Term::ANSIColor@3.2"  or something
>
> I would keep it simple like
>
> RDEPEND="$(perl-dep Term-ANSIColor)
>   $(perl-dep ">=Module-Load-0.240.0")"
>
> which would then translate into something like
>
> RDEPEND="( || ( ( >=perl-5.12 <=perl-5.18.1 ) perl-core/Term-ANSIColor )
>   || ( >=perl-5.18 >=perl-core/Module-Load-0.240.0 )"
>
> (or "|| ( ( perl-5.12* perl-5.14* ...
> || ( perl-5.18* ...)" -
> subject to further discussion)



The most annoying thing about that would be the implementation details.

The reason virtuals are such a nightmare for perl is really the way they're
oriented.

Every perl we release, we have to manually update "something", "somewhere"
in a location *other* than the perl ebuild itself.

Which means that instead of simply modifying perl's .ebuild, and walking
the contents of module::corelist and saying "This version provides X
version Y", one must instead have a way to reorient that data from the
perspective of dependencies.

That is, we must reorganize all the code in question in terms of X version
Y, as a list of things that provide that X version Y.

And thats the most no-trivial part of the equation.

The best solution I presently have for this problem, would be to have a
PROVIDES-${PV}.json file in every package under files/

That file would contain simply a list of provisions ( a literal string )
with a version of that provision.

And then we could mangle that data as an aggregate by iterating the whole
portage tree, collecting such files, and using them to provide the inverse
relationmap of "X verison Y is provided by Z version N", which could be
used by a suitable eclass.

Thats really as clean, and logical, as I can imagine making it. Just its
investment intensive, and investment intensive in ways that would be better
supported as a native portage feature.

-- 
Kent

[-- Attachment #2: Type: text/html, Size: 3136 bytes --]

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

On 29 September 2013 08:51, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com>wrote:

> The only reason anyone can make that claim is that no-one really knows
> what slot dictionaries are or how they'd work in practice. Until
> there's a rough description of how they work and a prototype
> implementation, "subslot dictionaries" is like "magic beans".
>

Indeed, but I can basically say without looking at such a competing
strategy, that as insane as the prototype may be in practise, it would be
grossly preferable to have that mechanic as an internal dependency control
mechanism, instead of something that more-or-less amounts to enumerating
the contents of a large package ( perl ), and essentially converting all
the file names in that package to a distinct USE_EXPAND useflag, and then
basically making them *ALL* on by default and having to tell users "Don't
turn them off unless you're smart/stupid!".

That'd be like shooting somebody in the head as a cure for cancer.


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1530 bytes --]

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Kent Fredric <kentfredric@gmail.com> wrote:
>>
>> 1. The virtual does not even exist :)
>>
>
> Nope, it does. Its just called =virtual/perl-Term-ANSIColor-4.20.0

In this case the virtual is buggy, becaues it does not contain
|| ( dev-lang/perl-5.18* ... )

Due to this, this particular example is not so ideal: I wanted to
have an example where the stable virtual/perl-... version is higher
than the one provided by (stable) perl. (This is then not the case
once the virtual got fixed and perl-5.18 gets stable.)
Please replace this example by one for which this is the case, say

virtual/perl-Digest-MD5

and assume that I want an ebuild for a package for which *any* version
of perl-Digest-MD5 would be sufficient.
In this (now hopefully correct) example, if I do not write
something like

RDEPEND="|| ( >=dev-lang/perl-5.12 virtual/perl-Digest-MD5 )"

or

RDEPEND="|| ( =dev-lang/perl-5.12* ... virtual/perl-Digest-MD5 )"

but instead only the "natural"

RDEPEND="virtual/perl-Digest-MD5"

I would currently pull in unnecessarily perl-core/Digest-MD5-2.520.0
or perl-core/Digest-MD5-2.530.0 (depending on whether the user
is using e.g. ~amd64 or amd64).

What I meant was that one might have the strange idea to use
as a "workaround" for this problem the dependency

RDEPEND="=virtual/perl-Digest-MD5-2.510.0-r2"

but this would be a very bad idea many reasons, in particular:

>> 2. It would collide with ebuilds depending on other versions.
>
> Nope, it would only collide in the case other ebuilds depended on a
> specific version

This is what I mean by "ebuilds depending on other versions".
The above "workaround" would unnecessarily block ebuilds which
contain

RDEPEND="... >=virtual/perl-Digest-MD5-2.520.0 "

> This is seen as a "feature"

I agree. I just wanted to point out why the "workaround" is not
really a workaround of the problem but would be just a very
bad idea.

>> 3. This version is only reasonable if perl-5.16 is the
>> perl version which the user has installed
>
> And when perl 5.18 hits tree [...] you'll
> need to change your dependency string in your package

Exactly: This is what you would have to do, and which is not solved
by the current virtual/... approach unless you accept pulling
in unneeded dependencies.

> Virtuals are much more convenient here

More convient than putting into the ebuild, but at the cost
not only of maintaining the virtuals but also of installing
duplicate packages on the user's system.
Take the above (hopefully correct) example that you
have the "natural"

RDEPEND="virtual/perl-Digest-MD5"

Although *any* perl version containing this module would be
sufficient, this dependency will install for a user with
unstable keywords in addition perl-core/Digest-MD5-2.330.0,
because this is the currently highest available version of
the virtual and not satisfied by any perl version.
(Similarly for the user with stable keywords, it will
install currently unnecessarily perl-core/Digest-MD5-2.320.0
and when perl-5.18 hits the stable tree, it is rather likely
that also >=virtual/Digest-MD5-2.330.0 is stable so that the
user has unnecessary duplicate versions installed throughout.)

> (Actually, thats a bug still, because corelist -a says 4.20.0 should be
> available in 5.18

In my local overlay, I have a list of ~30 more such virtual/perl-*
for which I have fixed dependencies (though maybe I have falsely
translated the version numbers). This does not belong to this
mailinglist, so if I you are interested in the virtual list or if
I should open a bug, drop me a pm.

Both suggested solutions (eclass or dev-lang/perl-... with useflags)
would avoid all of these problem:

You have to update only one list of packages for every perl version,
not update every single virtual/... and, more important, the
dependencies would not pull in unnecessary packages for the user.

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

On 29 September 2013 09:14, Martin Vaeth
<vaeth@mathematik.uni-wuerzburg.de>wrote:

> the
> dependencies would not pull in unnecessary packages for the user.
>

Its just every time you say "unnessecary", that also implies that users **
NEVER** want new versions of dependencies.

That is just not true for most of the gentoo tree. If a user doesn't want
"newer versions" of things, portage already provides them the tools to
solve that problem.

Now if a thing **in tree** explicitly doensn't work with a specific version
of something perl, then thats ground for declaring a blocker.

Its not grounds for perpetuating /*your*/ preference to not upgrade things
beyond core versions until other things **require** it to happen.

ie: Just because its not required that we have a newer version of something
in tree, thats not grounds for not providing it. Its simply grounds for end
users to exercise free will in deciding what they will and will not
install.


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1618 bytes --]

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

On 29 September 2013 09:14, Martin Vaeth
<vaeth@mathematik.uni-wuerzburg.de>wrote:

> this dependency will install for a user with
> unstable keywords
>

That, in itself, indicates the user is usually OK with "new versions of
things" ;)

corelist -a says virtual/perl-Digest-MD5-2.520.0 should || (  perl v5.18 )

Though that virtual is already stable, and as a result, will result in the
installation of that version of Digest::MD5 on perl versions <5.17

2.530.0 won't be in perl till 5.19+

One other reason you might want to consider that its *good* that we upgrade
things from perl to versions in perl-core/*.

CVEs. If a security hole is exposed in a version of something that is
shipped with perl, we can simply adjust the virtual and get it to pull in a
newer version via perl-core/*

Here, the "unnecessary" dependency could in fact be nessecary to avoid a
security hole in an older version that may be shipped with perl.

And in such a case, its "good" that installing foo, that depends on
"virtual/perl-SOMETHINGBROKEN" gets you a version more recent than in perl
itself.


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1830 bytes --]

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Kent Fredric <kentfredric@gmail.com> wrote:
>>
> So are you basically suggesting that for dual life modules, we simply
> ignore that they're dual-lifeable, and then when upstream splits a package
> from core so that its no longer dual life, that we simply ignore that too,
> and fake it still being core for the forseeable future?

Yes, this is the idea.

(Although I must admit that meanwhile I prefer the ebuild idea).

> IMO, really, the uses would be forced enabled for all users, because they
> should never be disabled. If they're part of the Perl built itself, they
> should get installed. Period.

Exactly: Those which are part of the Perl built itself would not have a
useflag at all.  Only for those which are *not* in part of the current
perl built you would have a useflag. For instance,
USE=Time-Piece would not occur in the perl-5.10* ebuilds but in all
other perl ebuilds in the form
PDEPEND="Time-Piece? ( perl-core/Time-Piece )"

Packages needing Time-Piece would then just
RDEPEND="dev-lang/perl[Time-Piece(+)]"
and would thus implicitly get the correct dependency.

Once the translation is done, the only place where maintenance is needed
are the perl-ebuilds.

> But why would you "depend on the perl-provided version"

Many packages only need the basic functionality of a library,
not a particular version. The problem is that something like

RDEPEND="virtual/perl-..."

implicitly depends on a particular version, namely the highest
one which is currently stable/unstable for your architecture
instead of the one which avoids unnecessary installation.

> Are you saying that, if something is provided by perl core, that we must
> never update to a cpan version?

If the user explicitly requires a particular package or
if a package explicitly depends on a certain minimal version,
the upgrade will be automatic.
However, if there is no need (dependency or explicit user request)
for a particular version, the installed version should be sufficient.

> You realise thats breaking how upstream thinks toolchains work right?

Upstream always thinks that you have to use the most current
unstable version of everything. This is not only true for perl
but also for every other bigger project (texlive or many others
come to mind).  It is the distribution's task to give a more
stable user experience.

> You're missing a very important point: Every single line of output without
> the uniq constraint is a package depending on a virtual.

This is not the case, but this need not be discussed here;
we agree that the number of *packages* depending on
virtual versions is large.

> The virtual is managing the need to have a conditional dependency based on
> the version of perl installed.
>
> You would need to ,without virtuals, modify *EVERY* ebuild containg a
> dependency on a virtual, to contain the respective conditional dependency
> enshrined in the virtual.

No. You just keep those few virtuals for which the packages explicitly
depend on the version. The remaining ones - for which the version
plays no role - are replaced *once* by the corresponding USE-dependency.

(However, as said above, I meanwhile prefer the eclass idea;
with the eclass you could treat both cases - those depending on
a particular version and those depending only on *some* version - in the
same way and even omit all virtuals.)

> Not sure about you, but the idea of modifying 20,000 lines of code instead
> of 1, is something I don't look on fondly.

Both solutions would require only *once* to modify the dependencies.
After the transfer is done, you only have to modify 1 line

>>|| ( >=dev-lang/perl-5.14 virtual/perl-Term-ANSIColor )
>> >
>> > That is plain wrong imo. You're prematurely optimising the dependency.
>>
>> The alternative is to pull in a duplicated installation which is
>> completely superfluous, since it is already installed by most
>> perl versions,
>
> You're not though. Thats the point. Virtuals are doing exactly that.
> They're just doing it in the virtual instead of your ebuild.

Only if there would be only *one* version of the corresponding
virtual. With *more* virtual versions as it is currently,
it is rather likely that an unneeded version of
perl-core/Term-ANSIColor is pulled it.

> If you are installing virtual Y version X, if you are on a version of Perl
> that contains Y version X, the virtual avoids the need to install the
> respective perl-core/X.

But if I am just installing virtual Y, version does not matter,
I will likely get virtual Y, versions X+Z although version X
is already there and thus version Z would not be needed.

> If you don't like that packages get updated, and pull newer-that-core
> versions of things, there's nothing to stop you grepping virtual/perl-*,
> and masking versions that don't match an "=your current version of perl".

This is a lazy excuse: Instead of producing proper dependencies
you are shifting the work to the user.
In fact, I did this work in the beginning, but not only is this a
lot of work for every perl upgrade (which is not so trivial to script),
you also need a lot of exceptions where the mask is not possible,
because some package needs a newer version.
For e.g. overlays which I provide to users, I do not want to
shift *my* work on the user's side and thus have to write clumsy
dependencies without virtual/...

If you use the ebuild or USE-idea you would get correct dependencies
which do not require the user to hack around and mask things only
to avoid unnecessary installations.

>> I'd also sooner consider attempting to eliminate the need for virtuals by
>> > unilaterally depending on perl-core/* , and vivifying perl-core/ from
>> > dev-lang/perl sources as needed.
>>
>> This breaks proper support for building/using binary packages for
>> perl-core/* since the installed files will depend on which packages
>> are installed at build-time.
>
> I'm not really sure what you're sayng here. Sorry :/

If you are writing an ebuild X whose action of src_install() depends
implicitly on the installed perl-version and the user executes
emerge --buildpkg=y X
on one machine and
emerge --usepkg X
on another machine (or at a later time) he will run into problems
without a warning if the perl versions of that two machines are not
the same (or have changed meanwhile) when the two commands are
executed.
Roughly speaking, you build an implicit dependency into the
*.tbz2 file which portage is not aware of and will not warn
the users about.

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Kent Fredric <kentfredric@gmail.com> wrote:
> --001a11336248343a2604e7770011
> Content-Type: text/plain; charset=UTF-8
>
> On 28 September 2013 23:36, Martin Vaeth
><vaeth@mathematik.uni-wuerzburg.de>wrote:
>
>>
>> Concerning the eclass idea which was already mentioned and
>> which is perhaps even better than my suggeestion from the
>> other posting, since it avoids some of the disadvantages:
>>
>> > by having an eclass that translates a special variable, say,
>> > PERL_MODULE_COREDEPS="Term::ANSIColor@3.2"  or something
>>
>> I would keep it simple like
>>
>> RDEPEND="$(perl-dep Term-ANSIColor)
>>   $(perl-dep ">=Module-Load-0.240.0")"

It would have been wiser to suggest here the syntax
perl-core/Term-ANSIColor and perl-core/Module-Load-0.240.0,
respectively so that the category need not be fixed to
"perl-core".

>>
>> which would then translate into something like
>>
>> RDEPEND="( || ( ( >=perl-5.12 <=perl-5.18.1 ) perl-core/Term-ANSIColor )
>>   || ( >=perl-5.18 >=perl-core/Module-Load-0.240.0 )"
>>
>> (or "|| ( ( perl-5.12* perl-5.14* ...
>> || ( perl-5.18* ...)" -
>> subject to further discussion)
>
>
> The most annoying thing about that would be the implementation details.

Why?  One of use probably misunderstand something:

> Every perl we release, we have to manually update "something", "somewhere"
> in a location *other* than the perl ebuild itself.

If you decide to use the eclass, then with each perl release
you would have to update the data in the eclass - namely the LIST of
provided packages for that new version. Something like

LIST_perl5.18=(
  perl-core/Term-ANSIColor-4.20
  perl-core/...-...
  dev-perl/...-...
  !perl-core/X-Y
  !!perl-core/X-Y
)
LIST_perl5.16=(
)
...
AVAILABLE_PERLS="5.18 5.16 ..."

where the special symbols ! means that version perl-core/X-Y is
not available in the tree and !! means that perl-core/X is not
available in the tree (in no version).
These LISTs are all which you have to update for new perl versions.
The rest should be handled by the logic of the eclass, and I do not
see why this should be hard to do:

1.
If perl-dep is called with the argument
  ">=category/package-version",
  ">category/package-version",
  "<=category/package-version",
  "<category/package-version",
  "=category/package-version", or
  "~category/package-version",
the function just output into || ( ... ) all perl versions which provide
a matching package according to LIST, followed by the passed argument
(the latter is skipped, if masked by ! or !! according to one LIST).
In case of "<=" or "<" in addition the blocker
"!>category/package-version" or "!>=category/package-version"
is output (unless masked by ! or !!).

2.
If perl-dep is called with the argument
  "catogory/package"
(without a version) the function just adds into || ( ... ) all
perl versions which provide the package and in addition ends with
the passed argument (unless masked by !!).

Implementation of that function is rather simple once you have
a version comparison function (I do not remember in the moment
whether this is already available in some eclass or will be
only available the next EAPI).
Of course, the output could be optimized by omitting the "|| ( )" symbols
if there is only one match etc.

> Which means that instead of simply modifying perl's .ebuild, and walking
> the contents of module::corelist and saying "This version provides X
> version Y", one must instead have a way to reorient that data from the
> perspective of dependencies.

Why?  You update the eclasses LIST variables in one place when a new
version of perl comes out. All functions needing a perl
package use the perl-dep function (which means a one-shot change for
all packages needing a perl package; for packages in the
tree this can be done by the "script" I mentioned).

> The best solution I presently have for this problem, would be to have
> a PROVIDES-${PV}.json file in every package under files/

Not under files but in the eclass, and the rest of the
work is done by the perl-dep function.

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Kent Fredric <kentfredric@gmail.com> wrote:
>
>> this dependency will install for a user with unstable keywords
>>
>
> That, in itself, indicates the user is usually OK with "new versions of
> things" ;)

You are intentionally confusing "new version" (AKA upgrade) with
_additional_ installation of a package, just because that package
contains a newer version.

If you explicitly installed that package, an upgrade of course
is desired, but *hard depending* on a package just because it
provides a newer version of a bundled package is more than
questionable:

Would you think that it is correct if e.g. a multimedia package
which *forcibly* has bundled ffmpeg should in addition *forcibly*
depend on the system ffmpeg library (for no other reason than
it is bundled anyway)? According to your definition of
"always guarantee to install new version" this would have to
be the case.

I agree that no solution is completely satisfactory:
The most correct solution might be to unbundle the library -
which for perl would mean to *not* install the provided
modules but put all of them in perl-core. But as often,
unbundling is here a *very* hard job (how to solve the
chicken-and-egg problem of installing perl packages
without having packages available for installation)
and probably manpower is missing to do this for every
perl version...

But in fact, this solution would allow complete
elimination of all artificial workarounds by
virtuals, eclasses or USE-flags and circumvent the problem
of duplicate installation of packages completely.

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1715 bytes --]

On 29 September 2013 11:13, Martin Vaeth
<vaeth@mathematik.uni-wuerzburg.de>wrote:

>
> > The best solution I presently have for this problem, would be to have
> > a PROVIDES-${PV}.json file in every package under files/
>
> Not under files but in the eclass, and the rest of the
> work is done by the perl-dep function.


The reason I suggested that approach, is because a list of modules and
versions that are contained within a specific version of a specific
package, is ostensibly a property of that version of that package.

And thus, with it in ${FILESDIR}, updating $P to a new $V makes it easy to
check for the existence of a respective ${FILESDIR}/PROVIDES-{$PV}.json

And thus, the technique can be used, not only for all of dev-lang/perl, but
for all of perl-core/* , and possibly even dev-perl/* , and maybe even
things outside these categories that happen to be providers of perl modules.

Then, the relative content in the eclass/  tree could be generated from
that data, and generated in the inverse form, allowing $(perl-dep "some
string here")  to have a unique mapping for all values of those strings, in
a form that was easy to decode and understand.

Thus, the conditionality of the dependency would not have to be determined
by messy bash code during the parsing of each and every ebuild, and would
instead be a relatively simple lookup table.

This moves all the overhead complexity from being something that only
happens in the developer side of things, reducing the number of ways the
eclass can fail.

And then we can also debate the qualities of specific ways of declaring the
dependency itself post-eclass, as it becomes a separate problem from how
the resolution is performed.

-- 
Kent

[-- Attachment #2: Type: text/html, Size: 2338 bytes --]

[gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Martin Vaeth]

Kent Fredric <kentfredric@gmail.com> wrote:
>>
>> > The best solution I presently have for this problem, would be to have
>> > a PROVIDES-${PV}.json file in every package under files/
>>
>> Not under files but in the eclass, and the rest of the
>> work is done by the perl-dep function.
>
> The reason I suggested that approach, is because a list of modules and
> versions that are contained within a specific version of a specific
> package, is ostensibly a property of that version of that package.

This is only one view of things. One might also have the view that
it is a property of the packages depending on it whether they accept
that dependency. Only the latter view is supported by the current way
dependencies work.

> And thus, with it in ${FILESDIR}, updating $P to a new $V makes it easy to
> check for the existence of a respective ${FILESDIR}/PROVIDES-{$PV}.json

Of course, if portage would support also the other view, it would be
nice, also from the user perspective:

For instance, if you have your home-brewn version of program X,
you can just install the version under its own package name Y and
make it satisfy all dependencies of X.
(Currently you have to mess around with package.provided which has
many drawbacks like e.g. problems with USE-dependencies and,
moreover, you cannot publish Y in an overlay without requiring
that the user of that overlay modifies his package.provided manually.)

However, I am afraid that this requires a rather fundamental rewrite
of the current dependency logic in package managers.

Re: [gentoo-dev] Re: [RFC] Policy for migrating library consumers to subslots

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1971 bytes --]

On 1 October 2013 04:52, Martin Vaeth <vaeth@mathematik.uni-wuerzburg.de>wrote:

>
> For instance, if you have your home-brewn version of program X,
> you can just install the version under its own package name Y and
> make it satisfy all dependencies of X.
> (Currently you have to mess around with package.provided which has
> many drawbacks like e.g. problems with USE-dependencies and,
> moreover, you cannot publish Y in an overlay without requiring
> that the user of that overlay modifies his package.provided manually.)


Here, this is even more annoying, because as long as this inverse
dependency is regulated by the eclass, even ebuilds in alternative trees
are out of luck from being seen as candidates for an multi-way provided
dependency, at least, you'd have to modify the eclass "somehow"

This is something a native PROVIDES= implementation is not limited by.

Though I reapeat, I do not want PROVIDES="perl-core/Foo-Bar", because thats
going to have us the same problem, as it declares provides only on the
granularity of *packages* not the granulatiry of files/modules.

I'd be championing something more like ( but not nessecarily exactly like )

PROVIDES="cpan-module://Module::Build/4.5"

Borrowing terms from the metadata.xml information, and the use of a
per-token-type protocol prefix allows for multiple unambiguous namespaces
that are not bound to the cat/package layout.

And here, the semantics after "://" could be implemented differently if
nessecary.

Consuming code would just say

DEPEND="cpan-module://Module::Build" or
DEPEND=">=cpan-module://Module::Build/4.0"

Though I don't know. The real problem is that as convenient as a PROVIDES=
interface may be, doing it performantly could be a nightmare, and portage
would probably need some sort of PROVIDES cache to even support such a
feature efficiently. ( Which is basically what my solution offers, just
with a developer-side-cache-generation instead of user-side-caching )

-- 
Kent

[-- Attachment #2: Type: text/html, Size: 2818 bytes --]