[gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Michał Górny]

We can basically assume PM merges either DEPEND or RDEPEND to that point
(depending on whether we're merging source ebuild or binary package).
---
 dependencies.tex |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/dependencies.tex b/dependencies.tex
index abcb820..e9c73d5 100644
--- a/dependencies.tex
+++ b/dependencies.tex
@@ -10,7 +10,8 @@
         \multicolumn{1}{c}{\textbf{Phase function}} &
         \multicolumn{1}{c}{\textbf{Satisfied dependency classes}} \\
         \midrule
-	\t{pkg\_pretend}, \t{pkg\_setup}, \t{pkg\_info}, \t{pkg\_nofetch} & None (ebuilds can rely only on the packages in the system set) \\
+	\t{pkg\_pretend}, \t{pkg\_info}, \t{pkg\_nofetch} & None (ebuilds can rely only on the packages in the system set) \\
+	\t{pkg\_setup} & packages common to \t{DEPEND} and \t{RDEPEND} \\
     \t{src\_unpack}, \t{src\_prepare}, \t{src\_configure}, \t{src\_compile}, \t{src\_test}, \t{src\_install} & \t{DEPEND} \\
     \t{pkg\_preinst}, \t{pkg\_postinst}, \t{pkg\_prerm}, \t{pkg\_postrm} & \t{RDEPEND} (unless the particular dependency results in a circular dependency, in which case it may be installed later) \\
 	\t{pkg\_config} & \t{RDEPEND}, \t{PDEPEND} \\
-- 
1.7.6

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

On Fri, 12 Aug 2011 20:03:37 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> We can basically assume PM merges either DEPEND or RDEPEND to that
> point (depending on whether we're merging source ebuild or binary
> package).

What if something's a binary (i.e. its DEPENDs aren't being installed)
and is being installed as part of an RDEPEND - RDEPEND cycle?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 624 bytes --]

On Sat, 13 Aug 2011 06:18:18 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Fri, 12 Aug 2011 20:03:37 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > We can basically assume PM merges either DEPEND or RDEPEND to that
> > point (depending on whether we're merging source ebuild or binary
> > package).
> 
> What if something's a binary (i.e. its DEPENDs aren't being installed)
> and is being installed as part of an RDEPEND - RDEPEND cycle?

We don't think that's even possible because RDEP+DEP cycle would be
unbreakable for the source ebuild.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

On Sat, 13 Aug 2011 09:33:48 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> On Sat, 13 Aug 2011 06:18:18 +0100
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > On Fri, 12 Aug 2011 20:03:37 +0200
> > Michał Górny <mgorny@gentoo.org> wrote:
> > > We can basically assume PM merges either DEPEND or RDEPEND to that
> > > point (depending on whether we're merging source ebuild or binary
> > > package).
> > 
> > What if something's a binary (i.e. its DEPENDs aren't being
> > installed) and is being installed as part of an RDEPEND - RDEPEND
> > cycle?
> 
> We don't think that's even possible because RDEP+DEP cycle would be
> unbreakable for the source ebuild.

It can be under a use flag, and thus be solvable.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Michał Górny]

We can basically assume PM merges either DEPEND or RDEPEND to that point
(depending on whether we're merging source ebuild or binary package).
---
 dependencies.tex |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/dependencies.tex b/dependencies.tex
index abcb820..557660a 100644
--- a/dependencies.tex
+++ b/dependencies.tex
@@ -10,7 +10,8 @@
         \multicolumn{1}{c}{\textbf{Phase function}} &
         \multicolumn{1}{c}{\textbf{Satisfied dependency classes}} \\
         \midrule
-	\t{pkg\_pretend}, \t{pkg\_setup}, \t{pkg\_info}, \t{pkg\_nofetch} & None (ebuilds can rely only on the packages in the system set) \\
+	\t{pkg\_pretend}, \t{pkg\_info}, \t{pkg\_nofetch} & None (ebuilds can rely only on the packages in the system set) \\
+	\t{pkg\_setup} & packages common to \t{DEPEND} and \t{RDEPEND} (but see below) \\
     \t{src\_unpack}, \t{src\_prepare}, \t{src\_configure}, \t{src\_compile}, \t{src\_test}, \t{src\_install} & \t{DEPEND} \\
     \t{pkg\_preinst}, \t{pkg\_postinst}, \t{pkg\_prerm}, \t{pkg\_postrm} & \t{RDEPEND} (unless the particular dependency results in a circular dependency, in which case it may be installed later) \\
 	\t{pkg\_config} & \t{RDEPEND}, \t{PDEPEND} \\
-- 
1.7.6

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 588 bytes --]

On Sat, 13 Aug 2011 10:14:50 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> We can basically assume PM merges either DEPEND or RDEPEND to that
> point (depending on whether we're merging source ebuild or binary
> package).

I think we're going about this in the wrong way. We should probably
remove all mention of circular dependencies, and just state that
there's nothing except system things guaranteed for pkg_*.

If there's a need for dependencies that will definitely be installed for
pkg_setup, we should introduce an IDEPEND (for 'install').

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ulrich Mueller]

>>>>> On Sat, 13 Aug 2011, Ciaran McCreesh wrote:

> I think we're going about this in the wrong way. We should probably
> remove all mention of circular dependencies, and just state that
> there's nothing except system things guaranteed for pkg_*.

Except that large parts of the tree rely on packages in RDEPEND being
available in pkg_*.

> If there's a need for dependencies that will definitely be installed
> for pkg_setup, we should introduce an IDEPEND (for 'install').

And then at some point we will have circular IDEPEND dependencies and
the package manager will have to break such cycles, as it does for
RDEPEND now.

Ulrich

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]

On Sat, 13 Aug 2011 10:38:28 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:
> >>>>> On Sat, 13 Aug 2011, Ciaran McCreesh wrote:
> > I think we're going about this in the wrong way. We should probably
> > remove all mention of circular dependencies, and just state that
> > there's nothing except system things guaranteed for pkg_*.
> 
> Except that large parts of the tree rely on packages in RDEPEND being
> available in pkg_*.

Then those packages are broken. (And we can't ban RDEPEND cycle
breaking, since large parts of the tree rely upon that too.)

> > If there's a need for dependencies that will definitely be installed
> > for pkg_setup, we should introduce an IDEPEND (for 'install').
> 
> And then at some point we will have circular IDEPEND dependencies and
> the package manager will have to break such cycles, as it does for
> RDEPEND now.

No, IDEPEND will simply not allow cycle breaking, in the same way as
DEPEND. IDEPEND would be, in effect, DEPEND that is also honoured for
binary packages.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ulrich Mueller]

>>>>> On Sat, 13 Aug 2011, Ciaran McCreesh wrote:

>> Except that large parts of the tree rely on packages in RDEPEND
>> being available in pkg_*.

> Then those packages are broken.

Welcome to reality. ;-)

>> > If there's a need for dependencies that will definitely be
>> > installed for pkg_setup, we should introduce an IDEPEND (for
>> > 'install').
>> 
>> And then at some point we will have circular IDEPEND dependencies
>> and the package manager will have to break such cycles, as it does
>> for RDEPEND now.

> No, IDEPEND will simply not allow cycle breaking, in the same way as
> DEPEND. IDEPEND would be, in effect, DEPEND that is also honoured
> for binary packages.

I'm not convinced. IDEPEND would cover a very specific usage case, and
IMHO it wouldn't be so different from an intersection of DEPEND and
RDEPEND. Not different enough to justify introduction of another
variable.

Maybe package managers could introduce it as an internal variable
(containing packages common to DEPEND and RDEPEND)? And in case that
we should at some point switch to an Exherbo-style syntax for
dependencies, then it could be specified explicitly?

Ulrich

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2283 bytes --]

On Sat, 13 Aug 2011 12:10:45 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:
> >> Except that large parts of the tree rely on packages in RDEPEND
> >> being available in pkg_*.
> 
> > Then those packages are broken.
> 
> Welcome to reality. ;-)

Reality is that RDEPEND cycle breaking happens, so those packages only
work if by fluke they're not in an RDEPEND cycle.

Now, we *could* weasel our way out of it by saying that it's illegal
for any repository to include a package that would induce such a cycle
between packages that rely upon the intersection of DEPEND and RDEPEND
being available in pkg_*. But that's pretty horrible, so if we do that
then we really need to fix things in future EAPIs.

> I'm not convinced. IDEPEND would cover a very specific usage case, and
> IMHO it wouldn't be so different from an intersection of DEPEND and
> RDEPEND.

The thing is, RDEPEND-RDEPEND cycles *have* to be breakable. There are
enough genuine instances in the tree that we can't simply turn RDEPEND
into "DEPEND for binaries".

> Not different enough to justify introduction of another variable.

Well... You say loads of packages in the tree rely upon it... So I'd
say it's rather a common use case. And making it explicit makes things
easy for developers, since it's fairly apparent that the subtleties of
the different *DEPENDs are lost on most people. "If you use it in pkg_*
it belongs in IDEPEND" is easy.

> Maybe package managers could introduce it as an internal variable
> (containing packages common to DEPEND and RDEPEND)?

That's a little fiddly and ill-specified... If DEPEND has cat/pkg and
RDEPEND has >=cat/pkg-1.23, are they common? What about foo? ( cat/pkg
) vs an unconditional cat/pkg?

> And in case that we should at some point switch to an Exherbo-style
> syntax for dependencies, then it could be specified explicitly?

I'm pretty much operating on the assumption that we'll have at least
SDEPEND in the next EAPI, and the embedded people have been after
compiled-against-DEPEND for five years or more, so I suspect we a
single var syntax of some kind or other... Whether it's just labels,
just annotations, mixed labels and annotations Exherbo-style or
something else, I don't know...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ulrich Mueller]

>>>>> On Sat, 13 Aug 2011, Ciaran McCreesh wrote:

> On Sat, 13 Aug 2011 12:10:45 +0200
> Ulrich Mueller <ulm@gentoo.org> wrote:
>> >> Except that large parts of the tree rely on packages in RDEPEND
>> >> being available in pkg_*.
>> 
>> > Then those packages are broken.
>> 
>> Welcome to reality. ;-)

> Reality is that RDEPEND cycle breaking happens, so those packages
> only work if by fluke they're not in an RDEPEND cycle.

> Now, we *could* weasel our way out of it by saying that it's illegal
> for any repository to include a package that would induce such a
> cycle between packages that rely upon the intersection of DEPEND and
> RDEPEND being available in pkg_*. But that's pretty horrible, so if
> we do that then we really need to fix things in future EAPIs.

I'd still suggest that for existing EAPIs we should go with mgorny's
latest patch, simply because "packages common to DEPEND and RDEPEND
(but see below)" is a more accurate description than "none".

However, we should keep in mind that the situation with respect to
pkg_* is not well defined. Therefore a new dependency type could be
beneficial for future EAPIs.

Ulrich

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

On Mon, 15 Aug 2011 08:12:55 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:
> I'd still suggest that for existing EAPIs we should go with mgorny's
> latest patch, simply because "packages common to DEPEND and RDEPEND
> (but see below)" is a more accurate description than "none".

But if we do that, people will claim it's the package mangler's fault
if packages common to DEPEND and RDEPEND aren't available. You're
asking for something unimplementable to be specified.

"none" is correct.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ulrich Mueller]

>>>>> On Mon, 15 Aug 2011, Ciaran McCreesh wrote:

> On Mon, 15 Aug 2011 08:12:55 +0200
> Ulrich Mueller <ulm@gentoo.org> wrote:
>> I'd still suggest that for existing EAPIs we should go with
>> mgorny's latest patch, simply because "packages common to DEPEND
>> and RDEPEND (but see below)" is a more accurate description than
>> "none".

> But if we do that, people will claim it's the package mangler's
> fault if packages common to DEPEND and RDEPEND aren't available.
> You're asking for something unimplementable to be specified.

No, they can't claim that, because we explicitly say that there are
restrictions in case of circular dependencies.

Ulrich

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

On Mon, 15 Aug 2011 09:09:04 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:
> >> I'd still suggest that for existing EAPIs we should go with
> >> mgorny's latest patch, simply because "packages common to DEPEND
> >> and RDEPEND (but see below)" is a more accurate description than
> >> "none".
> 
> > But if we do that, people will claim it's the package mangler's
> > fault if packages common to DEPEND and RDEPEND aren't available.
> > You're asking for something unimplementable to be specified.
> 
> No, they can't claim that, because we explicitly say that there are
> restrictions in case of circular dependencies.

The problem is, there are nearly always circular dependencies. Any
package that depends upon autotools is part of a hundred-plus-large
circular dependency cycle. So if we're saying packages can count on
RDEPENDs being merged beforehand except when necessary to resolve
cycles, then we need to say how cycles are to be resolved.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ulrich Mueller]

>>>>> On Mon, 15 Aug 2011, Ciaran McCreesh wrote:

> The problem is, there are nearly always circular dependencies. Any
> package that depends upon autotools is part of a hundred-plus-large
> circular dependency cycle. So if we're saying packages can count on
> RDEPENDs being merged beforehand except when necessary to resolve
> cycles, then we need to say how cycles are to be resolved.

Hm, I always thought that e.g. for packages that have no reverse
dependencies the issue would not arise. For example, if you have the
following RDEPEND cycle:

    C ← D ← E
    ↓   ↑
    B → A

then the ordering of A, B, C and D would be unspecified, but it would
be guaranteed that package E (which is not part of the cycle) is being
installed last.

Ulrich

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

On Mon, 15 Aug 2011 09:43:52 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:
> Hm, I always thought that e.g. for packages that have no reverse
> dependencies the issue would not arise. For example, if you have the
> following RDEPEND cycle:
> 
>     C ← D ← E
>     ↓   ↑
>     B → A
> 
> then the ordering of A, B, C and D would be unspecified, but it would
> be guaranteed that package E (which is not part of the cycle) is being
> installed last.

Nope. If you've just got RDEPENDs, there's no guarantee of any order at
all. If you had this:

     C ← D ← E <=== F
     ↓   ↑
     B → A

where F's dep is a DEPEND (and where F is not a binary package), then
you're guaranteed that all of A, B, C, D and E will be merged before F,
but you don't get any guarantee as to the order amongst A, B, C, D, E.

Introducing a stronger guarantee wouldn't even help: here, the
maintainer of E could carefully check that there is no possible cycle
containing both D and E, making it safe to use D in pkg_setup, and then
the next week A's maintainer (or worse, an overlay) could come along and
make A RDEPEND upon E. Then E's carefully checked pkg_setup would
suddenly become broken through something unrelated to either E or its
direct dependencies, and there would be no way to know about it until
people get hit by weird bugs.

The only fix is to tell people they must write pkg_* code to carry on
working if dependencies aren't present, and then to introduce IDEPEND
(or the equivalent label) in EAPI 5.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-pms] [PATCH] pkg_setup() can rely on packages common to DEPEND & RDEPEND.

[Ulrich Mueller]

>>>>> On Sat, 13 Aug 2011, Ciaran McCreesh wrote:

> On Fri, 12 Aug 2011 20:03:37 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
>> We can basically assume PM merges either DEPEND or RDEPEND to that
>> point (depending on whether we're merging source ebuild or binary
>> package).

> What if something's a binary (i.e. its DEPENDs aren't being installed)
> and is being installed as part of an RDEPEND - RDEPEND cycle?

Good point, looks like we've missed that case.

In fact, Michał's first wording included a clause covering circular
dependencies, but then I persuaded him that it wouldn't be needed: ;-)

<mgorny> http://dpaste.com/592914/ like this?
<mgorny> should be simplest words I can think of
<ulm> mgorny: why "but see below"?
<mgorny> RDEPEND
<mgorny> the long parenthesis there :P
[...]
<ulm> I think that case cannot occur becasue a circular dependency cannot be
      broken in a place where there's a DEPEND too
<mgorny> hm, true I think

Ulrich