[gentoo-dev] RFC: extending pkgmove (slotmove) actions to apply transitionally to ebuilds

[gentoo-dev] RFC: extending pkgmove (slotmove) actions to apply transitionally to ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3609 bytes --]

Hello, everyone.

I'd like to suggest extending the pkgmove (slotmove) action code
a little bit to reduce the amount of issues hit by users during
the transition period following package moves. Please read on
and let me know what you think.


The problem
-----------

Package moves (and slot moves) are currently a three part process:

1. committing the package under the new name (slot), and removing
   the old files,

2. updating all package references (*DEPEND, *_version, ...) in other
   ebuilds,

3. adding a pkgmove (slotmove) entry to apply the move on installed
   packages.

This design implies two significant problems:

a. with many packages needing changes, the process with CVS is split
extends in time and may end up causing users to try to use half-updated
package tree,

b. third-party repositories need to be updated manually following
the changes, and become broken before that is done.

While you may say that we can work-around (a) by stopping rsync updates
(which we usually don't care to do), and we can ignore (b) many users
are actually hitting issues. This has been proved again during
the recent eselect move.

The common issues include:

A. packages still depending on the old package have unsatisfiable
dependencies (if the old package was removed before updating the deps
-- e.g. third-party repositories),

B. file collisions due to PM attempting to install new-named package
alongside old one (users syncing before pkgmove is committed).


Solution
--------

In order to hide those issues from users and provide means for
painless migration, I'd like to propose that package (slot) moves are
applied not only to installed packages but also transparently to
ebuilds during dependency resolution.

This specifically means that:

1) All dependencies (and has_version, best_version calls) on the old
package name (slot) still work, though they get transparently converted
to match the new name (slot).

2) Developers are expected to commit the pkgmove (slotmove) as soon
as the new ebuild is committed, and can remove the old package (slot)
afterwards.

3) QA tools (repoman) complain about dependencies on old name/slot,
either using the regular dependency graph errors (i.e. omitting
the update-mapping) or new dedicated checks (additionally listing
the new name/slot).

Both issues are solved here:

A) Users of third-party repositories don't suffer because of package
moves -- since the repositories don't have to be updated immediately
after the package move.

B) Delays in committing package updates to Gentoo don't cause broken
dependency graphs -- therefore making it possible to perform updates
involving multiple packages in CVS painlessly.


Rationale
---------

The current package move logic already makes it impossible to use
the old name. The Package Managers don't keep track of updates applied,
and do re-apply the same update multiple times.

Therefore, any reference to the old name (slot) added after the update
will still get converted to the new name (slot) next time the updates
are applied. In particular, this makes it impossible to add blockers
on the old package name.

Since the old package name (slot) can no longer be used for any
purpose, we can safely map it implicitly to the new package name.
No new damage is done.

One can doubt that this will result in dependencies not being fixed.
However, forcing immediate breakage upon users is not the proper way
of enforcing QA. We have repoman and pcheck for that.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

[gentoo-dev] Re: RFC: extending pkgmove (slotmove) actions to apply transitionally to ebuilds

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1887 bytes --]

>>>>> On Sat, 4 Apr 2015, Michał Górny wrote:

> The problem
> -----------

> Package moves (and slot moves) are currently a three part process:

> 1. committing the package under the new name (slot), and removing
>    the old files,

Actually, you shouldn't remove the old package at this point because
it would break reverse dependencies.

> 2. updating all package references (*DEPEND, *_version, ...) in other
>    ebuilds,

> 3. adding a pkgmove (slotmove) entry to apply the move on installed
>    packages.

And only at this point the old package can be removed:
https://devmanual.gentoo.org/ebuild-writing/ebuild-maintenance/index.html#moving-ebuilds

However, all this will be moot once we have moved to git, because
atomic package moves will be possible then.

> [...]

> Rationale
> ---------

> The current package move logic already makes it impossible to use
> the old name. The Package Managers don't keep track of updates
> applied, and do re-apply the same update multiple times.

> Therefore, any reference to the old name (slot) added after the
> update will still get converted to the new name (slot) next time the
> updates are applied. In particular, this makes it impossible to add
> blockers on the old package name.

> Since the old package name (slot) can no longer be used for any
> purpose, we can safely map it implicitly to the new package name.
> No new damage is done.

This is not true for slotmoves. The previous slot can be reused by
versions not matching the dependency spec of the move. One can even
move some versions to a new slot, while leaving others in the old one.

For example, you could have app-misc/foo-1:0 and app-misc/foo-2:0 and
then do the following slotmove:

slotmove =app-misc/foo-2* 0 2

How would your transparent conversion treat >=app-misc/foo-1:0 in a
dependency?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: RFC: extending pkgmove (slotmove) actions to apply transitionally to ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2319 bytes --]

Dnia 2015-04-04, o godz. 20:14:21
Ulrich Mueller <ulm@gentoo.org> napisał(a):

> >>>>> On Sat, 4 Apr 2015, Michał Górny wrote:
> 
> > The problem
> > -----------
> 
> > Package moves (and slot moves) are currently a three part process:
> 
> > 1. committing the package under the new name (slot), and removing
> >    the old files,
> 
> Actually, you shouldn't remove the old package at this point because
> it would break reverse dependencies.

I wasn't listing the steps in order.

> 
> > 2. updating all package references (*DEPEND, *_version, ...) in other
> >    ebuilds,
> 
> > 3. adding a pkgmove (slotmove) entry to apply the move on installed
> >    packages.
> 
> And only at this point the old package can be removed:
> https://devmanual.gentoo.org/ebuild-writing/ebuild-maintenance/index.html#moving-ebuilds
> 
> However, all this will be moot once we have moved to git, because
> atomic package moves will be possible then.

No, the issue of third-party repositories will still be valid.

> > [...]
> 
> > Rationale
> > ---------
> 
> > The current package move logic already makes it impossible to use
> > the old name. The Package Managers don't keep track of updates
> > applied, and do re-apply the same update multiple times.
> 
> > Therefore, any reference to the old name (slot) added after the
> > update will still get converted to the new name (slot) next time the
> > updates are applied. In particular, this makes it impossible to add
> > blockers on the old package name.
> 
> > Since the old package name (slot) can no longer be used for any
> > purpose, we can safely map it implicitly to the new package name.
> > No new damage is done.
> 
> This is not true for slotmoves. The previous slot can be reused by
> versions not matching the dependency spec of the move. One can even
> move some versions to a new slot, while leaving others in the old one.
> 
> For example, you could have app-misc/foo-1:0 and app-misc/foo-2:0 and
> then do the following slotmove:
> 
> slotmove =app-misc/foo-2* 0 2
> 
> How would your transparent conversion treat >=app-misc/foo-1:0 in a
> dependency?

As far as I'm concerned, this is a hack and as such it doesn't have to
cover all the possible cases.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: RFC: extending pkgmove (slotmove) actions to apply transitionally to ebuilds

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1064 bytes --]

>>>>> On Sat, 4 Apr 2015, Michał Górny wrote:

>> This is not true for slotmoves. The previous slot can be reused by
>> versions not matching the dependency spec of the move. One can even
>> move some versions to a new slot, while leaving others in the old
>> one.
>> 
>> For example, you could have app-misc/foo-1:0 and app-misc/foo-2:0
>> and then do the following slotmove:
>> 
>> slotmove =app-misc/foo-2* 0 2
>> 
>> How would your transparent conversion treat >=app-misc/foo-1:0 in a
>> dependency?

> As far as I'm concerned, this is a hack and as such it doesn't have to
> cover all the possible cases.

But in the worst case, your "hack" can cause a broken dependency
graph. On the one hand, above mentioned >=app-misc/foo-1:0 matches
all versions affected by the slotmove, so it should be converted.
On the other hand, it is a perfectly valid dependency specification
which could have been added after the slotmove, in which case it
shouldn't be converted. You cannot know here what the intentions of
the developer are.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: RFC: extending pkgmove (slotmove) actions to apply transitionally to ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1299 bytes --]

Dnia 2015-04-04, o godz. 21:36:37
Ulrich Mueller <ulm@gentoo.org> napisał(a):

> >>>>> On Sat, 4 Apr 2015, Michał Górny wrote:
> 
> >> This is not true for slotmoves. The previous slot can be reused by
> >> versions not matching the dependency spec of the move. One can even
> >> move some versions to a new slot, while leaving others in the old
> >> one.
> >> 
> >> For example, you could have app-misc/foo-1:0 and app-misc/foo-2:0
> >> and then do the following slotmove:
> >> 
> >> slotmove =app-misc/foo-2* 0 2
> >> 
> >> How would your transparent conversion treat >=app-misc/foo-1:0 in a
> >> dependency?
> 
> > As far as I'm concerned, this is a hack and as such it doesn't have to
> > cover all the possible cases.
> 
> But in the worst case, your "hack" can cause a broken dependency
> graph. On the one hand, above mentioned >=app-misc/foo-1:0 matches
> all versions affected by the slotmove, so it should be converted.
> On the other hand, it is a perfectly valid dependency specification
> which could have been added after the slotmove, in which case it
> shouldn't be converted. You cannot know here what the intentions of
> the developer are.

But it *will* be converted in vdb the next time updates are applied.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: RFC: extending pkgmove (slotmove) actions to apply transitionally to ebuilds

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

>>>>> On Sat, 4 Apr 2015, Michał Górny wrote:

> Dnia 2015-04-04, o godz. 21:36:37
> Ulrich Mueller <ulm@gentoo.org> napisał(a):

>> But in the worst case, your "hack" can cause a broken dependency
>> graph. On the one hand, above mentioned >=app-misc/foo-1:0 matches
>> all versions affected by the slotmove, so it should be converted.
>> On the other hand, it is a perfectly valid dependency specification
>> which could have been added after the slotmove, in which case it
>> shouldn't be converted. You cannot know here what the intentions of
>> the developer are.

> But it *will* be converted in vdb the next time updates are applied.

IIUC, dependencies in the VDB will only be converted if the target of
the slotmove is installed on the user's system?

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]