[gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2930 bytes --]

Hello fellow developers

== Situation ==

When specifying a dependency like cat/pkg it will default to cat/pkg:*
which is defined in `man 5 ebuild` as:

    * Indicates  that  any  slot value is acceptable. In addition,
      for runtime dependencies, indicates that the package will not
      break if the matched package is uninstalled and replaced by a
      different matching package in a different slot.

This default reflects different behavior than what we use slots for,
besides allowing side-by-side installations we rather use it to
specifically depend on a new major version. (eg. dev-libs/glib).

Let's say I want to a add a new major version of cat/pkg to the Portage
tree, introducing it in the same SLOT="0" isn't an option. This gives
us two options, one is SLOT="2", the other is to create cat/pkg2 or so.

Creating a new SLOT is the most sane thing going forward; but, as the
default (:*) depends on any SLOT, this needs a half thousand commits to
fix up reverse dependencies. Thus, instead a new package is made. [1]

When our defaults force us down such path, that can't be good and it
affects the quality of our Portage tree; so, this makes me wonder, can
we change the default from :* to :0? What do you think?

 [1] https://bugs.gentoo.org/show_bug.cgi?id=493652
     "media-libs/libsdl2: should be a SLOT=2 of media-libs/libsdl"

== Task ==

If we agree we do this; in order to change :* to :0, we need to change
the PMS to cover this change and implement it in the package managers.

Before we do that, we need to evaluate how practical this is to apply.
While we are trying to fix the default behavior, what would changing
the default from :* to :0 break?

One thing that directly comes to mind is that dependencies that have no
SLOT="0" ebuild present would need us to manually specify a specific
SLOT; given that this is a not so common situation, the amount of
commits needed here is low.

Another thing that comes to mind is that we need to check what to do
with packages were the highest available version does not belong to
SLOT="0"; technically, restricting these to SLOT="0" will not cause
breakage, it might however cause some blockers. We'll have to look
closer into how we can alleviate this result.

Is there anything else we need to take into account?

== Summary ==

Situation:

    Defaulting to :* on SLOT-less dependencies makes it hard to add a
    new SLOT to a package, using :0 instead would not make this a
    problem and not require to fix up all reverse dependencies.

Task:

    Check results, correct some exceptions, update PMS, implement it.

Thank you in advance for participating in this discussion.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Patrick Lauer]

On 12/09/2013 12:54 AM, Tom Wijsman wrote:

> Creating a new SLOT is the most sane thing going forward; but, as the
> default (:*) depends on any SLOT, this needs a half thousand commits to
> fix up reverse dependencies. Thus, instead a new package is made. [1]


Pff. Lazy.



> When our defaults force us down such path, that can't be good and it
> affects the quality of our Portage tree; so, this makes me wonder, can
> we change the default from :* to :0? What do you think?

That just shifts the breakage to other people, who then have to do more
work.

> If we agree we do this; in order to change :* to :0, we need to change
> the PMS to cover this change and implement it in the package managers.
> 
> Before we do that, we need to evaluate how practical this is to apply.
> While we are trying to fix the default behavior, what would changing
> the default from :* to :0 break?
> 
> One thing that directly comes to mind is that dependencies that have no
> SLOT="0" ebuild present would need us to manually specify a specific
> SLOT; given that this is a not so common situation, the amount of
> commits needed here is low.

And now you make updating a lot more fun, because slotted packages need
to be explicitly changed if there's a new slot happening. Just to hide
your own laziness.

> Another thing that comes to mind is that we need to check what to do
> with packages were the highest available version does not belong to
> SLOT="0"; technically, restricting these to SLOT="0" will not cause
> breakage, it might however cause some blockers. We'll have to look
> closer into how we can alleviate this result.

Yup, bad idea.

500 commits vs. making things more complicated for everyone ... srsly?

Re: [gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2890 bytes --]

On Mon, 09 Dec 2013 07:57:34 +0800
Patrick Lauer <patrick@gentoo.org> wrote:

> On 12/09/2013 12:54 AM, Tom Wijsman wrote:
> 
> > Creating a new SLOT is the most sane thing going forward; but, as
> > the default (:*) depends on any SLOT, this needs a half thousand
> > commits to fix up reverse dependencies. Thus, instead a new package
> > is made. [1]
> 
> Pff. Lazy.

Yes, but who is lazy here? The person that wants to commit the
dependency or the people whom depend on the implicit :* behavior?

Or someone else?

> > When our defaults force us down such path, that can't be good and it
> > affects the quality of our Portage tree; so, this makes me wonder,
> > can we change the default from :* to :0? What do you think?
> 
> That just shifts the breakage to other people, who then have to do
> more work.

Doing a smaller bit of useful work to spare out tons of useless work.

As part of a new EAPI it doesn't break. Why do you think so?

Why would this yields more work? The dependencies need to be checked
anyway as port of version bumps; so, better do them right at once.

> > If we agree we do this; in order to change :* to :0, we need to
> > change the PMS to cover this change and implement it in the package
> > managers.
> > 
> > Before we do that, we need to evaluate how practical this is to
> > apply. While we are trying to fix the default behavior, what would
> > changing the default from :* to :0 break?
> > 
> > One thing that directly comes to mind is that dependencies that
> > have no SLOT="0" ebuild present would need us to manually specify a
> > specific SLOT; given that this is a not so common situation, the
> > amount of commits needed here is low.
> 
> And now you make updating a lot more fun, because slotted packages
> need to be explicitly changed if there's a new slot happening. Just
> to hide your own laziness.

As per my first question of this reply, whose laziness do you mean?

> > Another thing that comes to mind is that we need to check what to do
> > with packages were the highest available version does not belong to
> > SLOT="0"; technically, restricting these to SLOT="0" will not cause
> > breakage, it might however cause some blockers. We'll have to look
> > closer into how we can alleviate this result.
> 
> Yup, bad idea.

As part of a new EAPI the above is no longer necessary as the change
isn't done in place; furthermore, even if we don't do it as part
of a new EAPI repoman can cover this with a QA warning.

> 500 commits vs. making things more complicated for everyone ... srsly?

Why do you think this idea makes things more complicated for everyone?

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Rich Freeman]

On Sun, Dec 8, 2013 at 6:57 PM, Patrick Lauer <patrick@gentoo.org> wrote:
> On 12/09/2013 12:54 AM, Tom Wijsman wrote:
>>
>> One thing that directly comes to mind is that dependencies that have no
>> SLOT="0" ebuild present would need us to manually specify a specific
>> SLOT; given that this is a not so common situation, the amount of
>> commits needed here is low.
>
> And now you make updating a lot more fun, because slotted packages need
> to be explicitly changed if there's a new slot happening. Just to hide
> your own laziness.

Frankly, your tone is rude here.  Don't personalize things.  The goal
is to eliminate non-value-added work.  If a change makes work for
others that should be pointed out.  If a change saves somebody time,
that isn't "laziness."  Work isn't a virtue - accomplishing something
is.  Discuss the merits of the proposal, and don't debate the morals
of the person making the proposal.

Second, I'm not sure exactly what you're trying to say here.  Slotted
packages don't need to be changed under the proposed approach any more
than they otherwise would.  If you have SLOT=0 of libfoo and want to
introduce SLOT=1 it is the same amount of work for the libfoo
maintainer either way.  The only difference is that if half the tree
depends on libfoo and breaks with libfoo:1 then there are a bazillion
failures all the sudden.

If you meant that packages that depend on slotted packages need to be
changed, then that isn't really correct either.  If a package depends
on libfoo:0 instead of libfoo:*, it will work just fine when libfoo:1
is introduced - it will just continue to pull in libfoo:0.  Then over
time maintainers can test and add an || dep for libfoo:1 and the
package will work with either.

The status quo requires the libfoo maintainers to do a ton of work to
get everybody to change their deps before introducing a new slot.
Either that or they do what everybody else apparently does and not use
slots at all, just creating libfoo2.  Well, if you can't reliably
create new slots for dependencies, then what is the point of having a
syntax for slot dependencies in the first place?

On IRC there was some discussion and it may make sense to not make the
default :0 (or :0=), but rather to just not have any default in a
future EAPI - every single dependency line would therefore contain a
slot.  I'm not convinced wildcards should even be allowed - the whole
point of introducing new slots is that the new version may not be
compatible, so how can you say in advance that any slot is acceptable?
 If the new library is API-compatible then it should just be a new
subslot in the same slot, and if it is ABI-compatible then it should
be in the same subslot.  The whole point of subslots is to handle
packages that have API breaks, and I don't see how wildcards against
future versions not even written yet can be appropriate there.  The
only thing that it does is saves maintainers the trouble of looking up
what slots the package was actually tested against, which sounds a bit
like a certain word that starts with l...  :)

Rich

Re: [gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Sergey Popov]

[-- Attachment #1: Type: text/plain, Size: 3392 bytes --]

08.12.2013 20:54, Tom Wijsman пишет:
> Hello fellow developers
> 
> == Situation ==
> 
> When specifying a dependency like cat/pkg it will default to cat/pkg:*
> which is defined in `man 5 ebuild` as:
> 
>     * Indicates  that  any  slot value is acceptable. In addition,
>       for runtime dependencies, indicates that the package will not
>       break if the matched package is uninstalled and replaced by a
>       different matching package in a different slot.
> 
> This default reflects different behavior than what we use slots for,
> besides allowing side-by-side installations we rather use it to
> specifically depend on a new major version. (eg. dev-libs/glib).
> 
> Let's say I want to a add a new major version of cat/pkg to the Portage
> tree, introducing it in the same SLOT="0" isn't an option. This gives
> us two options, one is SLOT="2", the other is to create cat/pkg2 or so.
> 
> Creating a new SLOT is the most sane thing going forward; but, as the
> default (:*) depends on any SLOT, this needs a half thousand commits to
> fix up reverse dependencies. Thus, instead a new package is made. [1]
> 
> When our defaults force us down such path, that can't be good and it
> affects the quality of our Portage tree; so, this makes me wonder, can
> we change the default from :* to :0? What do you think?
> 
>  [1] https://bugs.gentoo.org/show_bug.cgi?id=493652
>      "media-libs/libsdl2: should be a SLOT=2 of media-libs/libsdl"
> 
> == Task ==
> 
> If we agree we do this; in order to change :* to :0, we need to change
> the PMS to cover this change and implement it in the package managers.
> 
> Before we do that, we need to evaluate how practical this is to apply.
> While we are trying to fix the default behavior, what would changing
> the default from :* to :0 break?
> 
> One thing that directly comes to mind is that dependencies that have no
> SLOT="0" ebuild present would need us to manually specify a specific
> SLOT; given that this is a not so common situation, the amount of
> commits needed here is low.
> 
> Another thing that comes to mind is that we need to check what to do
> with packages were the highest available version does not belong to
> SLOT="0"; technically, restricting these to SLOT="0" will not cause
> breakage, it might however cause some blockers. We'll have to look
> closer into how we can alleviate this result.
> 
> Is there anything else we need to take into account?
> 
> == Summary ==
> 
> Situation:
> 
>     Defaulting to :* on SLOT-less dependencies makes it hard to add a
>     new SLOT to a package, using :0 instead would not make this a
>     problem and not require to fix up all reverse dependencies.
> 
> Task:
> 
>     Check results, correct some exceptions, update PMS, implement it.
> 
> Thank you in advance for participating in this discussion.
> 

In short - please do NOT do this. More complicated answer - you break
the whole idea of slots. When user types 'emerge cat/foo' it means now -
"i want cat/foo, whatever versions it will be(depending on mask,
keywords, etc, etc)". Your proposal changes this behaviour drastically,
and reasons for such changes are not worth it.

-- 
Best regards, Sergey Popov
Gentoo developer
Gentoo Desktop Effects project lead
Gentoo Qt project lead
Gentoo Proxy maintainers project lead


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

Re: [gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 3306 bytes --]

On Mon, 09 Dec 2013 10:52:15 +0400
Sergey Popov <pinkbyte@gentoo.org> wrote:

> In short - please do NOT do this.

Why not? What about other solutions? What do you think?

There have come up several solutions, the one suggested in the first
mail is no longer of interest due to SLOT="0" no longer being special;
now ideas tend to focus towards disallowing slot-less dependencies, or
otherwise implement under the form of a repoman warning.

> More complicated answer - you break
> the whole idea of slots. When user types 'emerge cat/foo' it means
> now - "i want cat/foo, whatever versions it will be(depending on mask,
> keywords, etc, etc)". Your proposal changes this behaviour
> drastically, and reasons for such changes are not worth it.

A package ATOM used in another place warrants another discussion; when
an user types it, what does the user mean? We could opt to not change
the behavior for users if their usage is different.

Note that the syntax (and thus usage) is different by design:

     $ emerge -1pv '|| ( sys-apps/openrc sys-apps/systemd )'
    !!! '|| ( sys-apps/openrc sys-apps/systemd )' is not a valid
        package atom.
    !!! Please check ebuild(5) for full details.

The reasons for changing dependency syntax are worth it:

For the dependency syntax, having :* as a default breaks things or
causes a lot of work. If explicit slots (or :0) were the default, it
works and you spare out dealing with lots of reverse dependencies when
you introduce a new slot.

When you depend on libfoo; I think you mean to depend on the libfoo
that is currently in the Portage tree (eg. libfoo:0 or so), whereas
libfoo:* would break as soon as the libfoo maintainer starts a new slot.

With libfoo:*, this then results in the following procedure:

1. Add libfoo:2 to the Portage tree and package.mask.
2. Inform everyone to update the dependency accordingly, wait for it.
3. Unmask libfoo:2. (Somewhere in the future)

Whereas with an implicit or explicit libfoo:${SLOT}, this results in:

1. Add libfoo:2 to the Portage tree. (Right now, without mask)
2. Maintainers support libfoo:2 over time as they bump packages.

As the former is frustrating, it can have library maintainers create a
new package instead of a new SLOT or not add the new version at all;
whereas the latter would allow the library maintainer to just add it,
especially as it is the task of individual maintainers to test their
package against new libraries as they come along.

Thus, that's why I'd like to see slot-less dependencies go and make it
policy or at least a repoman QA warning that people specify them.

Specifying it without a slot is moving the work of dependency testing
on the library maintainer instead of the package maintainer, causing the
library maintainer to either test tons of packages or file tons of bug.

Why have the library maintainer do extra work when a package maintainer
can just test it instead during a version bump? (As required anyway)

What do you think? Why do you think we should still keep a default :*?

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Rich Freeman]

On Mon, Dec 9, 2013 at 5:55 AM, Tom Wijsman <TomWij@gentoo.org> wrote:
> For the dependency syntax, having :* as a default breaks things or
> causes a lot of work. If explicit slots (or :0) were the default, it
> works and you spare out dealing with lots of reverse dependencies when
> you introduce a new slot.

I was giving this some more thought and here is another way of looking at this.

Let A be the set of all situations where it makes sense to create a
new slot for a package that has reverse dependencies.

Let B be the subset of A where it makes sense for those reverse
dependencies to automatically use the new slot without any
intervention by the reverse dep maintainers.

If |B| >> |A \ B| then the current default makes sense.  (That is, if
the number of elements in B is much larger than the number of elements
in A that aren't in B.)

If |B| << |A \ B| then it would make sense to require all slot
dependencies to be explicit, and the ":*" dependency to be frowned
upon in general.

I think that the above just makes sense if you think about it, so I'll
use that as my initial premise for what follows.  By all means
challenge this.

So, what I'm going to do now is speculate that B = ∅ (that is B is the
empty set).  Therefore, it makes sense to get rid of the current
default and require all slot deps to be explicit.

If you think that B isn't the empty set, it is trivial for you to
demonstrate that this is the case.  Simply give me a single example of
a situation where:
1.  It makes sense for a dep to use a new slot.
2.  It makes sense for all of its reverse deps to automatically use
the new slot without any further intervention by the individual
reverse dep maintainers.

I can't think of any, and I'm all ears if somebody else can.  It seems
to me that our current default optimizes for a situation that is
dubious from a QA standpoint.  Its main virtue is that you don't have
to go look up what the current slot is for slot-less packages, but
honestly if you just stuck a :0 on every line of your ebuild chances
are it would work on the first install and at least the behavior would
be consistent for anybody else who uses it.

Rich

Re: [gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Ulrich Mueller]

>>>>> On Mon, 9 Dec 2013, Rich Freeman wrote:

> If you think that B isn't the empty set, it is trivial for you to
> demonstrate that this is the case.  Simply give me a single example of
> a situation where:
> 1.  It makes sense for a dep to use a new slot.
> 2.  It makes sense for all of its reverse deps to automatically use
> the new slot without any further intervention by the individual
> reverse dep maintainers.

app-editors/emacs, to start with.

If you go through the list of about 400 packages that have more than
one slot (out of 17000 packages in the portage tree), I'm sure you'll
find many more that fall in B. On first glance, only a minor part of
these 400 seem to be libraries.

Ulrich

Re: [gentoo-dev] Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2904 bytes --]

On Mon, 9 Dec 2013 17:19:34 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Mon, 9 Dec 2013, Rich Freeman wrote:
> 
> > If you think that B isn't the empty set, it is trivial for you to
> > demonstrate that this is the case.  Simply give me a single example
> > of a situation where:
> > 1.  It makes sense for a dep to use a new slot.
> > 2.  It makes sense for all of its reverse deps to automatically use
> > the new slot without any further intervention by the individual
> > reverse dep maintainers.
> 
> app-editors/emacs, to start with.
> 
> If you go through the list of about 400 packages that have more than
> one slot (out of 17000 packages in the portage tree), I'm sure you'll
> find many more that fall in B. On first glance, only a minor part of
> these 400 seem to be libraries.

It is surprising that there are only ~400 packages that have multiple
slots available in the Portage tree; trying to reproduce this, I get a
similar number. Out of a ~350 total, ~75 contain "libs/", ~100 contain
"dev-java/" (almost a third); there's are some ruby and haskell libs
too. These groups are almost all libraries, so it somewhat fifty-fifty. 

Note how Java packages already have a project policy to explicitly
specify slots on dependencies; as the eclass functionality relies on
that. As a consequence, it is easy to add a new slot to them. For the
Java herd it makes total sense to use explicit slots that way.

With new version bumps to reverse dependencies of dev-java libraries;
it is interesting to note that as time goes by, you will eventually
need to depend on the newer slot instead as upstream upgrades its
support for that dev-java library; dropping support for the old version.

But that's just a third and not necessarily representable, it can
however show how it works in practice.

Libraries' slot usage is harder to tell and needs a more thorough look
as to how these are done. It's easier to tell by someone whom has more
experience with how these libraries are versioned, seeing that most of
these are media-libs and x11-libs I'll try asking the relevant herds.

For applications (or should we say "all the rest") I think we need to
look at which kind of packages these are and what the slots mean.

For example, there are ~14 kernel sources that have multiple slots,
which are as far as I know not used as dependencies; it just serves
quite handy for the user to add it to the world file. There is also
www-client/google-chrome and sys-boot/grub, those are packages where
slots are meant for the user and not for reverse dependencies.

What do you think? Does someone have a different view? Please share. :)

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Martin Vaeth]

Tom Wijsman <TomWij@gentoo.org> wrote:
>
> It is surprising that there are only ~400 packages that have multiple
> slots available in the Portage tree; trying to reproduce this

No need to write a script for such things
(export EIX_LIMIT_COMPACT=0 if you do not pipe the output):
eix -c2

Packages with a slot different from "0" are found with eix -1,
i.e. the following lists packages for which a "0" default
would not lead to the desired result (because the only
provided slot is not "0"):
eix -c1 --and --not -2

Almost 1000 packages!

[gentoo-dev] Re: Dependencies default to accept any slot value acceptable (:*), can we default to :0 instead?

[Ulrich Mueller]

>>>>> On Tue, 10 Dec 2013, Martin Vaeth wrote:

> No need to write a script for such things
> (export EIX_LIMIT_COMPACT=0 if you do not pipe the output):
> eix -c2

$ eix -c2 | tail -n1
Found 473 matches.

> Packages with a slot different from "0" are found with eix -1,
> i.e. the following lists packages for which a "0" default
> would not lead to the desired result (because the only
> provided slot is not "0"):
> eix -c1 --and --not -2

> Almost 1000 packages!

That's all packages with a slot different from 0, which is larger than
the number of packages with more than one slot.

Ulrich