[gentoo-pms] Do we want an EAPI 5?

[gentoo-pms] Do we want an EAPI 5?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 659 bytes --]

Should we start pushing for a reasonably quick EAPI 5? I'd see it as
having:

* The stuff that was left out of EAPI 3/4, which is to say :=/:*
  dependencies, and the IUSE_IMPLICIT stuff (especially since right
  now people are breaking the rules and implicitly using 'prefix' when
  they shouldn't, and the rules for (+) and (-) are largely useless
  without the stricter control).

* Cleaning up some deprecated stuff (see recent bugs).

* A replacement for versionator, since apparently versionator is still
  using (a subset of) the ooooold version rules.

I think there was something else too, but I forget what...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] Do we want an EAPI 5?

[Sebastian Luther]

Am 30.06.2011 12:31, schrieb Ciaran McCreesh:
> Should we start pushing for a reasonably quick EAPI 5? I'd see it as
> having:
> 
> * The stuff that was left out of EAPI 3/4, which is to say :=/:*
>   dependencies, and the IUSE_IMPLICIT stuff (especially since right
>   now people are breaking the rules and implicitly using 'prefix' when
>   they shouldn't, and the rules for (+) and (-) are largely useless
>   without the stricter control).

You shouldn't insist on these two as long as there is no portage
implementation.

Are people (ebuild devs) really aware what introducing slot operator
deps would mean?
To make any use of them portage would have to stop updating installed
packages' metadata with ebuild metadata, which in turn means that
updating deps without revbump is going to cause problems for users.
I'm not saying that this is a bad thing, but it might not be what people
want.

Could you please give a summary (or point me to one) of the discussion
about :=/:*?
Specifically, why do we need two of them instead of declaring one of
them the default. And if we want both, what does it mean to not specify
one of them?

> 
> * Cleaning up some deprecated stuff (see recent bugs).
> 
> * A replacement for versionator, since apparently versionator is still
>   using (a subset of) the ooooold version rules.

++

> 
> I think there was something else too, but I forget what...
> 

Re: [gentoo-pms] Do we want an EAPI 5?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2245 bytes --]

On Thu, 30 Jun 2011 14:43:22 +0200
Sebastian Luther <SebastianLuther@gmx.de> wrote:
> Am 30.06.2011 12:31, schrieb Ciaran McCreesh:
> > Should we start pushing for a reasonably quick EAPI 5? I'd see it as
> > having:
> > 
> > * The stuff that was left out of EAPI 3/4, which is to say :=/:*
> >   dependencies, and the IUSE_IMPLICIT stuff (especially since right
> >   now people are breaking the rules and implicitly using 'prefix'
> > when they shouldn't, and the rules for (+) and (-) are largely
> > useless without the stricter control).
> 
> You shouldn't insist on these two as long as there is no portage
> implementation.

We need the IUSE_IMPLICIT stuff. The tree's already abusing use
dependencies in a way that can't be handled correctly by a package
mangler without it. We can't afford to continue having a broken tree
because of a major screwup caused by the Portage people not thinking
things through when they reduced the EAPI 4 feature set.

Also, Zac's said that if the Council approves it, he'll have that
feature done within a week.

> Are people (ebuild devs) really aware what introducing slot operator
> deps would mean?
> To make any use of them portage would have to stop updating installed
> packages' metadata with ebuild metadata, which in turn means that
> updating deps without revbump is going to cause problems for users.
> I'm not saying that this is a bad thing, but it might not be what
> people want.

Portage's behaviour is already broken there -- think what happens when
ebuilds get removed.

> Could you please give a summary (or point me to one) of the discussion
> about :=/:*?

See the original EAPI 3 discussion. It's all there.

> Specifically, why do we need two of them instead of declaring one of
> them the default. And if we want both, what does it mean to not
> specify one of them?

We need developers to be explicit. Neither behaviour is a sensible
default, since both commonly occur in practice. Developers must
carefully think through which they mean and then write the appropriate
dependency. It can't be determined automatically, and it's not safe to
assume that one particular behaviour is "probably" what's meant.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] Do we want an EAPI 5?

[Sebastian Luther]

Am 30.06.2011 19:22, schrieb Ciaran McCreesh:
> On Thu, 30 Jun 2011 14:43:22 +0200
> Sebastian Luther <SebastianLuther@gmx.de> wrote:
>> Am 30.06.2011 12:31, schrieb Ciaran McCreesh:
>>> Should we start pushing for a reasonably quick EAPI 5? I'd see it as
>>> having:
>>>
>>> * The stuff that was left out of EAPI 3/4, which is to say :=/:*
>>>   dependencies, and the IUSE_IMPLICIT stuff (especially since right
>>>   now people are breaking the rules and implicitly using 'prefix'
>>> when they shouldn't, and the rules for (+) and (-) are largely
>>> useless without the stricter control).
>>
>> You shouldn't insist on these two as long as there is no portage
>> implementation.
> 
> We need the IUSE_IMPLICIT stuff. The tree's already abusing use
> dependencies in a way that can't be handled correctly by a package
> mangler without it. We can't afford to continue having a broken tree
> because of a major screwup caused by the Portage people not thinking
> things through when they reduced the EAPI 4 feature set.
> 
> Also, Zac's said that if the Council approves it, he'll have that
> feature done within a week.

In this case, ignore me on this one.

> 
>> Are people (ebuild devs) really aware what introducing slot operator
>> deps would mean?
>> To make any use of them portage would have to stop updating installed
>> packages' metadata with ebuild metadata, which in turn means that
>> updating deps without revbump is going to cause problems for users.
>> I'm not saying that this is a bad thing, but it might not be what
>> people want.
> 
> Portage's behaviour is already broken there -- think what happens when
> ebuilds get removed.
> 

I know. I'm not opposed to this change, but the needed work flow change
for ebuild devs has to be communicated.

>> Could you please give a summary (or point me to one) of the discussion
>> about :=/:*?
> 
> See the original EAPI 3 discussion. It's all there.
> 
Yeah, the whole discussion is there, but not a summary. I don't have the
time to wade through all these mails.

>> Specifically, why do we need two of them instead of declaring one of
>> them the default. And if we want both, what does it mean to not
>> specify one of them?
> 
> We need developers to be explicit. Neither behaviour is a sensible
> default, since both commonly occur in practice. Developers must
> carefully think through which they mean and then write the appropriate
> dependency. It can't be determined automatically, and it's not safe to
> assume that one particular behaviour is "probably" what's meant.
> 

Isn't it desirable that different PMs handle the "no operator" case in
the same way (independently of the question of having one or both
operators available)?

Re: [gentoo-pms] Do we want an EAPI 5?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2368 bytes --]

On Thu, 30 Jun 2011 20:48:46 +0200
Sebastian Luther <SebastianLuther@gmx.de> wrote:
> > Portage's behaviour is already broken there -- think what happens
> > when ebuilds get removed.
> 
> I know. I'm not opposed to this change, but the needed work flow
> change for ebuild devs has to be communicated.

Shouldn't be a workflow change. It's already policy to do a revbump if
dependencies change.

> >> Could you please give a summary (or point me to one) of the
> >> discussion about :=/:*?
> > 
> > See the original EAPI 3 discussion. It's all there.
> > 
> Yeah, the whole discussion is there, but not a summary. I don't have
> the time to wade through all these mails.

Part of the reason EAPI 3 dragged on for so long was that rather than
reading the discussion, people would say "I've not read the entire
thread, but it seems to me that ...", and then the entire discussion
would have to be repeated all over again.

If you're just looking for a summary, not details: say a user has
cat/dep:1, cat/dep:2 and cat/dep:3 installed, and wants to uninstall
cat/dep:1 and cat/dep:2. Say there's cat/pkg installed which has a dep
upon cat/dep. Then there's no way for the package mangler to know
which cat/dep slots are still 'needed', and which can be safely
removed. Thus, to be safe, it has to assume that all three slots might
be needed.

Nearly all packages that dep upon a slotted dependent have one of two
behaviours: they're ok with any slot that matches the rest of the spec
(including switching at runtime), or they need the best slot that was
present at install time. The former is :*, the latter :=.

There are a few perverse cases that don't fit these. Those need special
fancy handling, and they're sufficiently fiddly that we shouldn't be
holding up solving the large number easy cases to deal with one or two
weird packages.

> Isn't it desirable that different PMs handle the "no operator" case in
> the same way (independently of the question of having one or both
> operators available)?

The problem is that every way of handling the "no operator" case is
wrong for many real situations. You can assume either "any slot" or
"best slot", both of which will allow packages to be removed unsafely,
or you can assume "all slots", which is excessively paranoid for many
packages.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] Do we want an EAPI 5?

[Sebastian Luther]

Am 01.07.2011 08:12, schrieb Ciaran McCreesh:
> On Thu, 30 Jun 2011 20:48:46 +0200
> Sebastian Luther <SebastianLuther@gmx.de> wrote:
>>> Portage's behaviour is already broken there -- think what happens
>>> when ebuilds get removed.
>>
>> I know. I'm not opposed to this change, but the needed work flow
>> change for ebuild devs has to be communicated.
> 
> Shouldn't be a workflow change. It's already policy to do a revbump if
> dependencies change.
> 

It is policy, but as you're probably aware of, it's not followed in may
cases, which in turn raises the question if people wouldn't prefer to
change the policy instead of their work flow (not that I would suggest
that).
Someone needs to ask them if you want that in EAPI5.

>>>> Could you please give a summary (or point me to one) of the
>>>> discussion about :=/:*?
>>>
>>> See the original EAPI 3 discussion. It's all there.
>>>
>> Yeah, the whole discussion is there, but not a summary. I don't have
>> the time to wade through all these mails.
> 
> Part of the reason EAPI 3 dragged on for so long was that rather than
> reading the discussion, people would say "I've not read the entire
> thread, but it seems to me that ...", and then the entire discussion
> would have to be repeated all over again.

I think part of the problem here is that those things aren't written
down as a glep (or some similar document). Imo every change that leads
to a non trivial amount of discussion should be written down in such a
way. Otherwise it's hard for people that missed the original discussion
to catch up and those that took part in this discussion get frustrated
because they have to repeat themselves again and again.
This should be discussed in another thread if someone is interested.

> 
> If you're just looking for a summary, not details: say a user has
> cat/dep:1, cat/dep:2 and cat/dep:3 installed, and wants to uninstall
> cat/dep:1 and cat/dep:2. Say there's cat/pkg installed which has a dep
> upon cat/dep. Then there's no way for the package mangler to know
> which cat/dep slots are still 'needed', and which can be safely
> removed. Thus, to be safe, it has to assume that all three slots might
> be needed.
> 
Or you do it like portage does it and assume the package works with any
slot (the :* case) and if it doesn't, declare that a bug of the package.
Now giving ebuild devs the := operator allows them to say "the package
insist on the slot it was build against".

> Nearly all packages that dep upon a slotted dependent have one of two
> behaviours: they're ok with any slot that matches the rest of the spec
> (including switching at runtime), or they need the best slot that was
> present at install time. The former is :*, the latter :=.
> 
> There are a few perverse cases that don't fit these. Those need special
> fancy handling, and they're sufficiently fiddly that we shouldn't be
> holding up solving the large number easy cases to deal with one or two
> weird packages.
> 
Agreed.

>> Isn't it desirable that different PMs handle the "no operator" case in
>> the same way (independently of the question of having one or both
>> operators available)?
> 
> The problem is that every way of handling the "no operator" case is
> wrong for many real situations. You can assume either "any slot" or
> "best slot", both of which will allow packages to be removed unsafely,
> or you can assume "all slots", which is excessively paranoid for many
> packages.
> 
Do you see a way for the PM to decide which behavior it should use on
per case basis?
If yes, than having both operators makes sense.
If no, the PM has to decide on one of the behaviors anyways. Why not
specify which one that is (see above)?

Re: [gentoo-pms] Do we want an EAPI 5?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3429 bytes --]

On Fri, 01 Jul 2011 09:45:34 +0200
Sebastian Luther <SebastianLuther@gmx.de> wrote:
> >> I know. I'm not opposed to this change, but the needed work flow
> >> change for ebuild devs has to be communicated.
> > 
> > Shouldn't be a workflow change. It's already policy to do a revbump
> > if dependencies change.
> 
> It is policy, but as you're probably aware of, it's not followed in
> may cases, which in turn raises the question if people wouldn't
> prefer to change the policy instead of their work flow (not that I
> would suggest that).

Anyone not following the policy is already breaking things, and we
can't switch to the "use the tree ebuild" behaviour since it doesn't
work when ebuilds get removed. All we're doing is making an existing
bug more visible.

> I think part of the problem here is that those things aren't written
> down as a glep (or some similar document). Imo every change that leads
> to a non trivial amount of discussion should be written down in such a
> way. Otherwise it's hard for people that missed the original
> discussion to catch up and those that took part in this discussion
> get frustrated because they have to repeat themselves again and again.
> This should be discussed in another thread if someone is interested.

It's not worth it. It's a huge amount of work to summarise every
incorrect argument and false lead. It's not done by official standards
bodies, who have far more resources than we do, so what you're asking
for is just obstructionism and red tape.

> > If you're just looking for a summary, not details: say a user has
> > cat/dep:1, cat/dep:2 and cat/dep:3 installed, and wants to uninstall
> > cat/dep:1 and cat/dep:2. Say there's cat/pkg installed which has a
> > dep upon cat/dep. Then there's no way for the package mangler to
> > know which cat/dep slots are still 'needed', and which can be safely
> > removed. Thus, to be safe, it has to assume that all three slots
> > might be needed.
> > 
> Or you do it like portage does it and assume the package works with
> any slot (the :* case) and if it doesn't, declare that a bug of the
> package. Now giving ebuild devs the := operator allows them to say
> "the package insist on the slot it was build against".

Can't. Right now there's no way for packages to specify those kinds of
dependencies correctly. Assuming :* just isn't safe, and doesn't match
the historical meaning of lack-of-slot dependencies.

> > The problem is that every way of handling the "no operator" case is
> > wrong for many real situations. You can assume either "any slot" or
> > "best slot", both of which will allow packages to be removed
> > unsafely, or you can assume "all slots", which is excessively
> > paranoid for many packages.
> > 
> Do you see a way for the PM to decide which behavior it should use on
> per case basis?

The package mangler has to be told explicitly. It can't work it out
itself.

> If yes, than having both operators makes sense.
> If no, the PM has to decide on one of the behaviors anyways. Why not
> specify which one that is (see above)?

Having both operators provides a way of checking that developers have
explicitly confirmed which behaviour they mean. Without them, we don't
know whether no operator means "whichever default we decide", or
whether it means "I didn't realise that this dep has slots now".

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] Do we want an EAPI 5?

[Sebastian Luther]

Am 01.07.2011 10:09, schrieb Ciaran McCreesh:
> On Fri, 01 Jul 2011 09:45:34 +0200
> Sebastian Luther <SebastianLuther@gmx.de> wrote:
>>>> I know. I'm not opposed to this change, but the needed work flow
>>>> change for ebuild devs has to be communicated.
>>>
>>> Shouldn't be a workflow change. It's already policy to do a revbump
>>> if dependencies change.
>>
>> It is policy, but as you're probably aware of, it's not followed in
>> may cases, which in turn raises the question if people wouldn't
>> prefer to change the policy instead of their work flow (not that I
>> would suggest that).
> 
> Anyone not following the policy is already breaking things, and we
> can't switch to the "use the tree ebuild" behaviour since it doesn't
> work when ebuilds get removed. All we're doing is making an existing
> bug more visible.
The "make the bug more visible part" is what I'm up to.

> 
>> I think part of the problem here is that those things aren't written
>> down as a glep (or some similar document). Imo every change that leads
>> to a non trivial amount of discussion should be written down in such a
>> way. Otherwise it's hard for people that missed the original
>> discussion to catch up and those that took part in this discussion
>> get frustrated because they have to repeat themselves again and again.
>> This should be discussed in another thread if someone is interested.
> 
> It's not worth it. It's a huge amount of work to summarise every
> incorrect argument and false lead. It's not done by official standards
> bodies, who have far more resources than we do, so what you're asking
> for is just obstructionism and red tape.
> 
I'm not asking for a summary of everything someone said about the topic.
I'm asking for something between this and a PMS text, simply put, the
stuff that is mandated by the glep rules.

>>> If you're just looking for a summary, not details: say a user has
>>> cat/dep:1, cat/dep:2 and cat/dep:3 installed, and wants to uninstall
>>> cat/dep:1 and cat/dep:2. Say there's cat/pkg installed which has a
>>> dep upon cat/dep. Then there's no way for the package mangler to
>>> know which cat/dep slots are still 'needed', and which can be safely
>>> removed. Thus, to be safe, it has to assume that all three slots
>>> might be needed.
>>>
>> Or you do it like portage does it and assume the package works with
>> any slot (the :* case) and if it doesn't, declare that a bug of the
>> package. Now giving ebuild devs the := operator allows them to say
>> "the package insist on the slot it was build against".
> 
> Can't. Right now there's no way for packages to specify those kinds of
> dependencies correctly. Assuming :* just isn't safe, and doesn't match
> the historical meaning of lack-of-slot dependencies.
> 
I don't know what you mean with historical meaning. As long as I use
gentoo (since ~mid 2007) it has been that way. I agree that it isn't
safe, but that's how portage does it.
For EAPI5, there is the possibility to make this the default and leave
:= for the cases that don't behave this way (see below).

>>> The problem is that every way of handling the "no operator" case is
>>> wrong for many real situations. You can assume either "any slot" or
>>> "best slot", both of which will allow packages to be removed
>>> unsafely, or you can assume "all slots", which is excessively
>>> paranoid for many packages.
>>>
>> Do you see a way for the PM to decide which behavior it should use on
>> per case basis?
> 
> The package mangler has to be told explicitly. It can't work it out
> itself.
> 
>> If yes, than having both operators makes sense.
>> If no, the PM has to decide on one of the behaviors anyways. Why not
>> specify which one that is (see above)?
> 
> Having both operators provides a way of checking that developers have
> explicitly confirmed which behaviour they mean. Without them, we don't
> know whether no operator means "whichever default we decide", or
> whether it means "I didn't realise that this dep has slots now".
> 
Right, that's lost when a default is selected.

It now boils down to the question what people want.
a) The proposal that had been voted into EAPI3.
b) What I said above.

The first having the advantage that it's possible to check if a dep has
been updated to slot deps or not.

The latter has the advantage that it's simpler in the sense that it
introduces only one new thing (the := operator) to do one new thing,
namely to turn the :* behavior most people (those that use portage) are
used to into the := behavior. It also doesn't leave any uncertainty
about what the "no-operator" case means.

Re: [gentoo-pms] Do we want an EAPI 5?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3518 bytes --]

On Fri, 01 Jul 2011 10:54:59 +0200
Sebastian Luther <SebastianLuther@gmx.de> wrote:
> > Anyone not following the policy is already breaking things, and we
> > can't switch to the "use the tree ebuild" behaviour since it doesn't
> > work when ebuilds get removed. All we're doing is making an existing
> > bug more visible.
>
> The "make the bug more visible part" is what I'm up to.

A bug is a bug, even if people don't see it very frequently. Making
bugs visible is a *good* thing, since it means they're easier to
identify and fix.

> > It's not worth it. It's a huge amount of work to summarise every
> > incorrect argument and false lead. It's not done by official
> > standards bodies, who have far more resources than we do, so what
> > you're asking for is just obstructionism and red tape.
>
> I'm not asking for a summary of everything someone said about the
> topic. I'm asking for something between this and a PMS text, simply
> put, the stuff that is mandated by the glep rules.

Too much work. You're welcome to do it if you think it's necessary, but
Gentoo doesn't have the manpower to hold everything up for someone to
copy a bunch of largely irrelevant discussion into a document for every
little thing. The spec describes how it works; all the false starts are
no longer an issue.

> >> Or you do it like portage does it and assume the package works with
> >> any slot (the :* case) and if it doesn't, declare that a bug of the
> >> package. Now giving ebuild devs the := operator allows them to say
> >> "the package insist on the slot it was build against".
> > 
> > Can't. Right now there's no way for packages to specify those kinds
> > of dependencies correctly. Assuming :* just isn't safe, and doesn't
> > match the historical meaning of lack-of-slot dependencies.
> > 
> I don't know what you mean with historical meaning. As long as I use
> gentoo (since ~mid 2007) it has been that way. I agree that it isn't
> safe, but that's how portage does it.

Portage has never done full safety checking or purges, though -- partly
because doing so right now requires either being excessively paranoid
or not catching most unsafe operations.

> For EAPI5, there is the possibility to make this the default and leave
> := for the cases that don't behave this way (see below).

But then we won't know whether people really mean that (they usually
don't) or whether they just forgot. Given that developers aren't in the
habit of giving the least bit of thought to slots, we need a way for
repoman to know whether or not it needs to tell developers to pay
attention. Slot operator dependencies are most useful when they're
widely used.

> The latter has the advantage that it's simpler in the sense that it
> introduces only one new thing (the := operator) to do one new thing,
> namely to turn the :* behavior most people (those that use portage)
> are used to into the := behavior. It also doesn't leave any
> uncertainty about what the "no-operator" case means.

There's no uncertainty with the "no operator" case with the
Council-approved version either -- it means the developer has forgotten
to specify a behaviour, so the package manager should be paranoid and
assume it matches *all* slots since it can't prove that it doesn't.
That's not what Portage does currently; Portage's current behaviour
isn't strong enough to provide useful protection.

In any case, this is all rehashing existing discussion.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]