[gentoo-dev] EAPI 7 in Portage needs YOU!

[gentoo-dev] EAPI 7 in Portage needs YOU!

[Michał Górny]

Hi, everyone.

The time for EAPI 7 cutoff is coming near, and as usual it looks like we're going to scratch the hardest bits due to lack of implementation. Most notably, the following two features are unlikely to be implemented:

1. Runtime-switchable USE flags,

2. ||= (binding any-of) dep groups.

If you want to see either of them in EAPI 7, you need to start working on implementing them in Portage *right now*. I can help you comprehend the spec and Zac can give you some implementation tips but you have to do the coding yourself.

That said, help with other EAPI 7 items as well as reviews and testing of the existing code are also welcome.

The combined work of all developers involved in the implementation as well as complete change list and linked documentation can be found in the following pull request:

https://github.com/gentoo/portage/pull/208
-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Anthony G. Basile]

On 2/19/18 5:49 AM, Michał Górny wrote:

> 1. Runtime-switchable USE flags,

I need to understand this.  Where are the specs?

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Luigi Mantellini]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]

See this bug https://bugs.gentoo.org/424283

Speaking as a monkey user, I would like to have a fast dependency tree
resolution on emerge tasks.
Another exotic feature is to use a layered filesystem in order to keep
different sets on different layers. (Example: layer system: core utils,
layer desktop: application, layer user: other stuffs). Something like nixos
or coreos (rip.).  When install, portage will create a snapshot of the
destination layer, and install on this. This will increase the complexity
and will us happy to discover something of new.

ciao

luigi



On Mon, Feb 19, 2018 at 12:00 PM, Anthony G. Basile <blueness@gentoo.org>
wrote:

> On 2/19/18 5:49 AM, Michał Górny wrote:
>
> > 1. Runtime-switchable USE flags,
>
> I need to understand this.  Where are the specs?
>
> --
> Anthony G. Basile, Ph.D.
> Gentoo Linux Developer [Hardened]
> E-Mail    : blueness@gentoo.org
> GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
> GnuPG ID  : F52D4BBA
>
>


-- 
Luigi 'Comio' Mantellini
R&D - Software
Industrie Dial Face S.p.A.
Via Canzo, 4
20068 Peschiera Borromeo (MI), Italy

Tel.: +39 02 5167 2813
Fax: +39 02 5167 2459
web: www.idf-hit.com
mail: luigi.mantellini@idf-hit.com

[-- Attachment #2: Type: text/html, Size: 2048 bytes --]

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Michał Górny]

W dniu pon, 19.02.2018 o godzinie 06∶00 -0500, użytkownik Anthony G.
Basile napisał:
> On 2/19/18 5:49 AM, Michał Górny wrote:
> 
> > 1. Runtime-switchable USE flags,
> 
> I need to understand this.  Where are the specs?

https://dev.gentoo.org/~ulm/pms/7-draft/pms.html#x1-670007.3

Long story short, IUSE_RUNTIME specifies extra USE flags that do not
require rebuilding the package and whose status can be updated in place
(provided that the necessary dependencies are installed).

Implementation-wise, I think you would need to find the place where
Portage decides to rebuild package because of use flag changes (either
due to deps, --changed-use or --newuse).  You'd have to check if all
changed flags belong in IUSE_RUNTIME and if that's the case (and there
is no other reason to rebuild), use a new 'update-in-place' state
instead of 'rebuild'.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Michael Lienhardt]

Il 19/02/2018 11:49, Michał Górny ha scritto:
> 2. ||= (binding any-of) dep groups.

I don't understand what this group means, and the PMS-7 is unclear as well:
  "binding-any-of A binding-any-of group, which has the same format as the any-of group, but begins with the string ||= instead."
Is it a "or", like the "any-of" group, but with a different behavior at compiling/linking time?

Michael Lienhardt

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 513 bytes --]

>>>>> On Mon, 19 Feb 2018, Michael Lienhardt wrote:

>> 2. ||= (binding any-of) dep groups.

> I don't understand what this group means, and the PMS-7 is unclear as well:
>   "binding-any-of A binding-any-of group, which has the same format
>   as the any-of group, but begins with the string ||= instead."
> Is it a "or", like the "any-of" group, but with a different behavior
> at compiling/linking time?

It is explained in section 8.2.4:
https://dev.gentoo.org/~ulm/pms/7-draft/pms.html#x1-800008.2.4

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Mart Raudsepp]

On Mon, 2018-02-19 at 18:34 +0100, Ulrich Mueller wrote:
> > > > > > On Mon, 19 Feb 2018, Michael Lienhardt wrote:
> > > 2. ||= (binding any-of) dep groups.
> > I don't understand what this group means, and the PMS-7 is unclear
> > as well:
> >   "binding-any-of A binding-any-of group, which has the same format
> >   as the any-of group, but begins with the string ||= instead."
> > Is it a "or", like the "any-of" group, but with a different
> > behavior
> > at compiling/linking time?
> 
> It is explained in section 8.2.4:
> https://dev.gentoo.org/~ulm/pms/7-draft/pms.html#x1-800008.2.4

Maybe I missed this, but a real world use case example would be nice,
maybe someone feels a harder itch to scratch then :)

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Michał Górny]

W dniu pon, 19.02.2018 o godzinie 21∶32 +0200, użytkownik Mart Raudsepp
napisał:
> On Mon, 2018-02-19 at 18:34 +0100, Ulrich Mueller wrote:
> > > > > > > On Mon, 19 Feb 2018, Michael Lienhardt wrote:
> > > > 
> > > > 2. ||= (binding any-of) dep groups.
> > > 
> > > I don't understand what this group means, and the PMS-7 is unclear
> > > as well:
> > >   "binding-any-of A binding-any-of group, which has the same format
> > >   as the any-of group, but begins with the string ||= instead."
> > > Is it a "or", like the "any-of" group, but with a different
> > > behavior
> > > at compiling/linking time?
> > 
> > It is explained in section 8.2.4:
> > https://dev.gentoo.org/~ulm/pms/7-draft/pms.html#x1-800008.2.4
> 
> Maybe I missed this, but a real world use case example would be nice,
> maybe someone feels a harder itch to scratch then :)
> 

The original use case was for providers-like thingies, e.g.:

  ||= ( ffmpeg:0= libav:0= )

That said, I'd personally prefer doing that with proper USE_EXPAND
and REQUIRED_USE enforcing but this has been rejected.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1383 bytes --]

On 02/19/2018 11:38 AM, Michał Górny wrote:
> W dniu pon, 19.02.2018 o godzinie 21∶32 +0200, użytkownik Mart Raudsepp
> napisał:
>> On Mon, 2018-02-19 at 18:34 +0100, Ulrich Mueller wrote:
>>>>>>>> On Mon, 19 Feb 2018, Michael Lienhardt wrote:
>>>>>
>>>>> 2. ||= (binding any-of) dep groups.
>>>>
>>>> I don't understand what this group means, and the PMS-7 is unclear
>>>> as well:
>>>>   "binding-any-of A binding-any-of group, which has the same format
>>>>   as the any-of group, but begins with the string ||= instead."
>>>> Is it a "or", like the "any-of" group, but with a different
>>>> behavior
>>>> at compiling/linking time?
>>>
>>> It is explained in section 8.2.4:
>>> https://dev.gentoo.org/~ulm/pms/7-draft/pms.html#x1-800008.2.4
>>
>> Maybe I missed this, but a real world use case example would be nice,
>> maybe someone feels a harder itch to scratch then :)
>>
> 
> The original use case was for providers-like thingies, e.g.:
> 
>   ||= ( ffmpeg:0= libav:0= )
> 
> That said, I'd personally prefer doing that with proper USE_EXPAND
> and REQUIRED_USE enforcing but this has been rejected.

Indeed, the tendency of || deps to make things less deterministic can be
troublesome. Maybe automatic REQUIRED_USE satisfaction will make the
REQUIRED_USE approach more acceptable to those who have rejected it.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Michael Lienhardt]

Il 19/02/2018 20:38, Michał Górny ha scritto:
> W dniu pon, 19.02.2018 o godzinie 21∶32 +0200, użytkownik Mart Raudsepp
> napisał:
>> On Mon, 2018-02-19 at 18:34 +0100, Ulrich Mueller wrote:
>>> It is explained in section 8.2.4:
>>> https://dev.gentoo.org/~ulm/pms/7-draft/pms.html#x1-800008.2.4
>>
>> Maybe I missed this, but a real world use case example would be nice,
>> maybe someone feels a harder itch to scratch then :)
>>
> 
> The original use case was for providers-like thingies, e.g.:
> 
>    ||= ( ffmpeg:0= libav:0= )
> 
> That said, I'd personally prefer doing that with proper USE_EXPAND
> and REQUIRED_USE enforcing but this has been rejected.
> 

So, if I understand correctly, the ||= group is an "or" that must be resolved in the same way in the DEPEND and RDEPEND dependencies, right?

The documentation does not specify how this group interacts between different ebuilds.
I guess there is no interaction, but just to be sure, let consider the following corner-case example:
  - a package A has RDEPEND=||= ( ffmpeg:0= libav:0= ) while another package B has DEPEND=ffmpeg
  - the solver choose libav to solve the dependency of the first package and ffmpeg to solve the second, removing ffmpeg afterward
  - will package A break?

Michael Lienhardt

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Ian Stakenvicius]

[-- Attachment #1.1: Type: text/plain, Size: 3178 bytes --]

On 2018-02-19 07:19 PM, Michael Lienhardt wrote:
> 
> 
> Il 19/02/2018 20:38, Michał Górny ha scritto:
>> W dniu pon, 19.02.2018 o godzinie 21∶32 +0200, użytkownik Mart Raudsepp
>> napisał:
>>> On Mon, 2018-02-19 at 18:34 +0100, Ulrich Mueller wrote:
>>>> It is explained in section 8.2.4:
>>>> https://dev.gentoo.org/~ulm/pms/7-draft/pms.html#x1-800008.2.4
>>>
>>> Maybe I missed this, but a real world use case example would be nice,
>>> maybe someone feels a harder itch to scratch then :)
>>>
>>
>> The original use case was for providers-like thingies, e.g.:
>>
>>    ||= ( ffmpeg:0= libav:0= )
>>
>> That said, I'd personally prefer doing that with proper USE_EXPAND
>> and REQUIRED_USE enforcing but this has been rejected.
>>
> 
> So, if I understand correctly, the ||= group is an "or" that must be
> resolved in the same way in the DEPEND and RDEPEND dependencies, right?
> 
> The documentation does not specify how this group interacts between
> different ebuilds.
> I guess there is no interaction, but just to be sure, let consider the
> following corner-case example:
>  - a package A has RDEPEND=||= ( ffmpeg:0= libav:0= ) while another
> package B has DEPEND=ffmpeg
>  - the solver choose libav to solve the dependency of the first
> package and ffmpeg to solve the second, removing ffmpeg afterward
>  - will package A break?
> 
> Michael Lienhardt


I don't think interaction with other ebuilds is necessarily important,
or at least, not any different as what occurs with a package manager
decides to do something with the bound atom.  Using your example of
pkg-A built with libav:0= bound:

1- ffmpeg is merged (lets ignore that ffmpeg blocks libav for now) --
this wouldn't change anything regarding pkg-A as it is currently
merged, but if pkg-A is re-emerged for whatever reason then I believe
the package manager needs to decide according to the rules to build
against ffmpeg.

2- libav is unmerged -- pkg-A is broken.  It's likely up to the PM
implementation what to do here.  In the case of portage, and the
libav-unmerge being a soft-blocker, I believe the behaviour would be
to trigger a re-emerge of pkg-A which would then through standard
dependency resolution decide that it will now depend on ffmpeg.    An
alternative would be to upgrade the soft-blocker to a hard-blocker
since pkg-A is bound to libav:0= and so it can't be resolved
automatically.  A manual unmerge likely should trigger something about
pkg-A too but even in portage's case the user can do that and break
dependencies, so it's not obvious to me what would be done.

Now, since ffmpeg and libav do block eachother, the switch from one to
another would trigger both of these cases which IMO would mean portage
would therefore rebuild pkg-A after ffmpeg is emerged (and libav would
be unmerged before ffmpeg is merged).  And the multi-ebuild
interaction that you speak of would simply force ffmpeg as a hard
dependency in resolution, triggering the soft-block and therefore the
rebuild.

EAPI7 is TL;DR for me right now but I assume that it doesn't specify a
required action for "package broken"?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 248 bytes --]

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Ian Stakenvicius]

[-- Attachment #1.1: Type: text/plain, Size: 1228 bytes --]

On 2018-02-19 12:34 PM, Ulrich Mueller wrote:
>>>>>> On Mon, 19 Feb 2018, Michael Lienhardt wrote:
> 
>>> 2. ||= (binding any-of) dep groups.
> 
>> I don't understand what this group means, and the PMS-7 is
>> unclear as well: "binding-any-of A binding-any-of group, which
>> has the same format as the any-of group, but begins with the
>> string ||= instead." Is it a "or", like the "any-of" group, but
>> with a different behavior at compiling/linking time?
> 
> It is explained in section 8.2.4: 
> https://dev.gentoo.org/~ulm/pms/7-draft/pms.html#x1-800008.2.4
> 
> Ulrich
> 


Could we get some clarification/confirmation on this language?:


> In addition, for runtime dependencies, indicates that the package
> will break unless a matching package corresponding to the first
> immediate child element (in order of listing) installed as a
> build-time (DEPEND) dependency is available


...if I am reading this right, this means that the list of atoms is
iterated through first->last, and as soon as one is found to be
available at build-time, it chosen as the one that this package is
bound to and after that the package will be considered broken if it
doesn't exist/is later removed?



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 248 bytes --]

Re: [gentoo-dev] EAPI 7 in Portage needs YOU!

[Michał Górny]

Dnia 20 lutego 2018 03:30:10 CET, Ian Stakenvicius <axs@gentoo.org> napisał(a):
>On 2018-02-19 12:34 PM, Ulrich Mueller wrote:
>>>>>>> On Mon, 19 Feb 2018, Michael Lienhardt wrote:
>> 
>>>> 2. ||= (binding any-of) dep groups.
>> 
>>> I don't understand what this group means, and the PMS-7 is
>>> unclear as well: "binding-any-of A binding-any-of group, which
>>> has the same format as the any-of group, but begins with the
>>> string ||= instead." Is it a "or", like the "any-of" group, but
>>> with a different behavior at compiling/linking time?
>> 
>> It is explained in section 8.2.4: 
>> https://dev.gentoo.org/~ulm/pms/7-draft/pms.html#x1-800008.2.4
>> 
>> Ulrich
>> 
>
>
>Could we get some clarification/confirmation on this language?:
>
>
>> In addition, for runtime dependencies, indicates that the package
>> will break unless a matching package corresponding to the first
>> immediate child element (in order of listing) installed as a
>> build-time (DEPEND) dependency is available
>
>
>...if I am reading this right, this means that the list of atoms is
>iterated through first->last, and as soon as one is found to be
>available at build-time, it chosen as the one that this package is
>bound to and after that the package will be considered broken if it
>doesn't exist/is later removed?

Yes. Let me expand on that. Given a dep:

  ||= ( p[0] p[1] p[2]... )

p[i] with lower i has higher priority.

Implementation-wise, at the very minimum:

1. PM needs to ensure that at least one of p[i] is installed (doesn't matter which one).

2. PM needs to seek through p[i] at build time and record the p[i] with lowest i that was satisfied somehow (with explicit indication how it was obtained).

3. PM needs to treat this p[i] as unconditional (but weak-ish) dep, i.e. 'emerge --depclean' must not remove it even if p[j], j!=i is present.

4. Also, it needs to handle missing all p[i] in a non-crashing fashion, i.e. without requiring user to fix it manually.

However, if we want it to do more then locking deps forever, we should also consider the following (with i!=j):

1. If p[i] is locked, and p[j] is requested and blocking p[i], package should be rebuilt after installing p[j] and removing p[i]. 

2. If p[i] is locked, and p[j] gets installed where j<i (no blockers), PM may want to rebuild pkg to unlock p[i] (this should probably be controlled by option).

3. If p[i] is locked, and p[j] is installed where j>i (no blockers), PM should provide some way of uninstalling p[i] (and rebuilding pkg afterwards). My idea was to allow blocker syntax for emerge, i.e. "emerge -1v !dev-libs/foo " which would uninstall it like a blocker and rebuild all pkgs with weak deps.

-- 
Best regards,
Michał Górny (by phone)