[gentoo-dev] Unified DEPENDENCIES concept

[gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 4683 bytes --]

Since DEPENDENCIES hasn't been written up in a Gentoo-friendly manner,
and since the Exherbo documentation doesn't seem to suffice to explain
the idea here, here's some more details on the DEPENDENCIES proposal.

We observe that a typical package will have something like this:

    DEPEND="
        cat/first
        cat/second
        cat/third
        cat/fourth
        foo? (
            foo/one
            >=foo/two-2.34[monkeys]
        )
        bar? (
             bar/one
        )
        baz? (
             baz/one
             baz/two
        )
        fnord? (
             fnord/one
        )"

    RDEPEND="
        cat/second
        cat/third
        cat/fourth
        cat/fifth
        foo? (
            >=foo/two-2.34[monkeys]
            foo/three
        )
        bar? (
             bar/one
             bar/two
             bar/three
             bar/four
        )"

Note how many dependencies that we think of as being "the same" are
listed in two places, and are not in any way "the same".

Note also how the foo-related things, the bar-related things etc cannot
be grouped together by their fooness or barness, but are rather grouped
by their DEPENDness and RDEPENDness.

Right now we've just got three variables. In a not too distant EAPI, it
will be considerably more than three. We also have duplication; with
HDEPEND, this will sometimes become triplication.

Here's the important bit, which I shall prefix with some stars:

*** The point of DEPENDENCIES is not to replace n variables with one
*** variable.

*** The point of DEPENDENCIES is not to provide syntactic sugar over a
*** group of n variables.

*** The point of DEPENDENCIES is to allow a package's dependencies,
*** which can be very complicated, to be expressed accurately and
*** simply, in a way friendly to how developers deal with dependencies
*** in practice.

So here's what DEPENDENCIES solves:

Firstly, it allows developers to group together foo-related dependencies
and bar-related dependencies by their fooness and barness, not by their
role.

Secondly, it reduces error-prone duplication.

Thirdly, it avoids some fairly obtuse names (BADEPEND? LDEPEND?) in
favour of something a tad more readable.

Here's the other important bit, with more stars:

*** It does it by replacing the concept of "a package has build
*** dependencies, run dependencies, etc" with "a package has
*** dependencies, and each dependency is applicable at one or more of
*** build time, run tme, etc".

We could just mark each dependency individually by when it's
applicable. But that turns out to be quite verbose, since packages have
fairly long lists of dependencies of the same kind. So here's a much
more concise syntax:

    DEPENDENCIES="
        build:
            cat/first
        build+run:
            cat/second
            cat/third
            cat/fourth
        run:
            cat/fifth

        foo? (
            build:
                foo/one
            build+run:
                >=foo/two-2.34[monkeys]
            run:
                foo/three
        )

        bar? (
             build+run:
                 bar/one
             run:
                 bar/two
                 bar/three
                 bar/four
        )

        build:
            baz? (
                 baz/one
                 baz/two
            )

            fnord? (
                 fnord/one
            )"

Here are the rules for working out which labels apply to a given spec:

We have a "labels stack" when parsing. Initially, the value "build+run"
is on the top. Whenever we encounter a block ("( )", "use? ( )" etc),
we duplicate the top value of the stack, and when we leave the block we
pop the value off. Whenever we encounter labels, we replace the top
value on the stack with those labels. Whenever we encounter a spec, its
associated labels are those on the top of the stack.

To avoid weirdness, so long as we're inside a || block, use of labels
is forbidden. Note that this is a reflection of the current ambiguity
as to what DEPEND="|| ( a b )" RDEPEND="|| ( a b )" means (can you
build against a, and run against b?), and if we're wanting to tidy that
up, that should be an independent proposal.

These rules are mostly for flexibility: if developers sometimes prefer
to group things by role rather than by fooness, then that's possible
too.

The rules for eclass merging need changing too, to add a ( ) around
values rather than merely appending. This is a technicality, and isn't
developer visible.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 7759 bytes --]

On Fri, 7 Sep 2012 12:45:59 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> Since DEPENDENCIES hasn't been written up in a Gentoo-friendly manner,
> and since the Exherbo documentation doesn't seem to suffice to explain
> the idea here, here's some more details on the DEPENDENCIES proposal.
> 
> We observe that a typical package will have something like this:
> 
>     DEPEND="
>         cat/first
>         cat/second
>         cat/third
>         cat/fourth
>         foo? (
>             foo/one
>             >=foo/two-2.34[monkeys]
>         )
>         bar? (
>              bar/one
>         )
>         baz? (
>              baz/one
>              baz/two
>         )
>         fnord? (
>              fnord/one
>         )"
> 
>     RDEPEND="
>         cat/second
>         cat/third
>         cat/fourth
>         cat/fifth
>         foo? (
>             >=foo/two-2.34[monkeys]
>             foo/three
>         )
>         bar? (
>              bar/one
>              bar/two
>              bar/three
>              bar/four
>         )"

Now please express that using bash variable references instead of
the overly verbose expansions. Also, real examples are much more useful
than counting and monkeys. This is not mathematics.

> Note how many dependencies that we think of as being "the same" are
> listed in two places, and are not in any way "the same".
> 
> Note also how the foo-related things, the bar-related things etc
> cannot be grouped together by their fooness or barness, but are
> rather grouped by their DEPENDness and RDEPENDness.

What is fooness and barness? Because I got lost near the four
non-existent word.

> Right now we've just got three variables. In a not too distant EAPI,
> it will be considerably more than three. We also have duplication;
> with HDEPEND, this will sometimes become triplication.

Sometimes != always. Sometimes HDEPEND will actually simplify
the current state. By moving a few things from DEPEND to HDEPEND, we
may be actually able to remove COMMON_DEPEND. Sometimes.

> Here's the important bit, which I shall prefix with some stars:
> 
> *** The point of DEPENDENCIES is not to replace n variables with one
> *** variable.

Yes, it is.

> *** The point of DEPENDENCIES is not to provide syntactic sugar over a
> *** group of n variables.

Yes, it is.

> *** The point of DEPENDENCIES is to allow a package's dependencies,
> *** which can be very complicated, to be expressed accurately and
> *** simply, in a way friendly to how developers deal with dependencies
> *** in practice.

Didn't we already prove that the syntactic sugar DEPENDENCIES allows
can make the dependency specifications very unclear and not anywhere
close to simple?

> So here's what DEPENDENCIES solves:
> 
> Firstly, it allows developers to group together foo-related
> dependencies and bar-related dependencies by their fooness and
> barness, not by their role.
> 
> Secondly, it reduces error-prone duplication.

No, it doesn't. It just provides an alternate solution over how we do
that now. Does variable substitution ring a bell?

> Thirdly, it avoids some fairly obtuse names (BADEPEND? LDEPEND?) in
> favour of something a tad more readable.

That is a side effect. DEPENDENCIES doesn't disallow us from choosing
to use labels like 'd:', 'r:'. Similarly, bash variables don't anyhow
stand in the way of using more verbose names.

> Here's the other important bit, with more stars:
> 
> *** It does it by replacing the concept of "a package has build
> *** dependencies, run dependencies, etc" with "a package has
> *** dependencies, and each dependency is applicable at one or more of
> *** build time, run tme, etc".

Really?

I think it actually gives concept of 'a package has a few random groups
of dependencies, where each group is applicable at one or more of build
time, run time, etc and consists of one or more dependencies or groups
of dependencies, where each group is applicable at one or more of build
time, run time, etc (which may or may not be the same as parent group)
and consists of one or more dependencies or groups of dependencies,
where each group is applicable at one or more of build time, run time,
etc (which may or may not be the same as parent group) and consists of
one or more dependencies or groups of dependencies, where each group is
applicable at one or more of build time, run time, etc (which may or
may not be the same as parent group) and consists of one or more
dependencies or groups of dependencies, where each group is applicable
at one or more of build time, run time, etc (which may or may not be
the same as parent group) and consists of one or more dependencies or
groups of dependencies, where each group is applicable at one or more
of build time, run time, etc (which may or may not be the same as
parent group) and consists ...'

> We could just mark each dependency individually by when it's
> applicable. But that turns out to be quite verbose, since packages
> have fairly long lists of dependencies of the same kind. So here's a
> much more concise syntax:
> 
>     DEPENDENCIES="
>         build:
>             cat/first
>         build+run:
>             cat/second
>             cat/third
>             cat/fourth
>         run:
>             cat/fifth
> 
>         foo? (
>             build:
>                 foo/one
>             build+run:
>                 >=foo/two-2.34[monkeys]
>             run:
>                 foo/three
>         )
> 
>         bar? (
>              build+run:
>                  bar/one
>              run:
>                  bar/two
>                  bar/three
>                  bar/four
>         )
> 
>         build:
>             baz? (
>                  baz/one
>                  baz/two
>             )
> 
>             fnord? (
>                  fnord/one
>             )"
> 
> Here are the rules for working out which labels apply to a given spec:
> 
> We have a "labels stack" when parsing. Initially, the value
> "build+run" is on the top. Whenever we encounter a block ("( )",
> "use? ( )" etc), we duplicate the top value of the stack, and when we
> leave the block we pop the value off. Whenever we encounter labels,
> we replace the top value on the stack with those labels. Whenever we
> encounter a spec, its associated labels are those on the top of the
> stack.

These are the rules for a machine. People don't actually read
dependencies sequentially. Provide a good algorithm which works from
any position.

> To avoid weirdness, so long as we're inside a || block, use of labels
> is forbidden. Note that this is a reflection of the current ambiguity
> as to what DEPEND="|| ( a b )" RDEPEND="|| ( a b )" means (can you
> build against a, and run against b?), and if we're wanting to tidy
> that up, that should be an independent proposal.

Yes, that's something you could work on instead of throwing a lot of
work on us just to reinvent the wheel your way because you believe that
your bicycle is much better when it has square wheels.

> These rules are mostly for flexibility: if developers sometimes prefer
> to group things by role rather than by fooness, then that's possible
> too.
> 
> The rules for eclass merging need changing too, to add a ( ) around
> values rather than merely appending. This is a technicality, and isn't
> developer visible.

The algorithm you presented above was technicality as well.

So, now that your forked the thread shall we repeat all
the disadvantages which you love to forgot to mention?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

On Fri, 7 Sep 2012 14:29:41 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> > Here's the important bit, which I shall prefix with some stars:
> > 
> > *** The point of DEPENDENCIES is not to replace n variables with one
> > *** variable.
> 
> Yes, it is.

You've clearly either completely missed the point of all of this,
despite it being pointed out to you, or are just trying to cause
trouble. Whilst you may be thinking of a DEPENDENCIES proposal that is
about replacing n variables with one variable (and I've never seen such
a proposal being made), what we're discussing here is not that. Kindly
stay out of this discussion until you understand what it is we're
discussing.

> These are the rules for a machine. People don't actually read
> dependencies sequentially. Provide a good algorithm which works from
> any position.

Read backwards from the current position until you find a label. It's
the same algorithm you use to find out what the current section is when
you're reading an article.

> The algorithm you presented above was technicality as well.

Agreed, but it was explicitly requested by Brian, so I provided it.

> So, now that your forked the thread

...as requested by Brian...

> shall we repeat all the disadvantages which you love to forgot to
> mention?

No, please stay out of this until you understand the original proposal,
and in particular the parts marked with ***. So long as you don't
understand what we're discussing, I'm afraid any contributions you make
will simply be shouting and waving.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1104 bytes --]

On Fri, 7 Sep 2012 13:36:05 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> > These are the rules for a machine. People don't actually read
> > dependencies sequentially. Provide a good algorithm which works from
> > any position.
> 
> Read backwards from the current position until you find a label. It's
> the same algorithm you use to find out what the current section is
> when you're reading an article.

No, it doesn't work that way.

See:

  foo? (
    label:
      dev-foo/bar
  )
  dev-bar/foo <-

Reading backwards, I find 'label'.
  
> > shall we repeat all the disadvantages which you love to forgot to
> > mention?
> 
> No, please stay out of this until you understand the original
> proposal, and in particular the parts marked with ***. So long as you
> don't understand what we're discussing, I'm afraid any contributions
> you make will simply be shouting and waving.

What have parts marked with '***' to do with the disadvantages we have
already proved and which you didn't even bother answering?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/09/12 07:45 AM, Ciaran McCreesh wrote:
> [ Snip! ] Note also how the foo-related things, the bar-related
> things etc cannot be grouped together by their fooness or barness,
> but are rather grouped by their DEPENDness and RDEPENDness.
> 
> [ Snip! ]
> 
> So here's what DEPENDENCIES solves:
> 
> Firstly, it allows developers to group together foo-related
> dependencies and bar-related dependencies by their fooness and
> barness, not by their role. [ Snip! ] *** It does it by replacing
> the concept of "a package has build *** dependencies, run
> dependencies, etc" with "a package has *** dependencies, and each
> dependency is applicable at one or more of *** build time, run tme,
> etc".

And this is the specific point that I don't like about DEPENDENCIES
versus *DEPEND.  As a developer, I personally find it much more
straight-forward to fill in the deps needed for each role, rather than
specifying the role(s) that each dep will play a part in.

Although I realize that technically I could still do that (have the
dep list be role-centric rather than dep-centric), given that the
point of this change is (as stated above) to organize deps the other
way, I can't really get behind the idea.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBKCcAACgkQ2ugaI38ACPDARgD+Inqa+/o1kTqlfuf7gC6wa3Da
YAmj/F7Glno1QuzALboA/1l/XCbTr27XBGv7ULcvN0rdqqrBmarA8CgsySQiZRUB
=Xwaz
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]

On Fri, 7 Sep 2012 16:23:16 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> On Fri, 7 Sep 2012 13:36:05 +0100
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > > These are the rules for a machine. People don't actually read
> > > dependencies sequentially. Provide a good algorithm which works
> > > from any position.
> > 
> > Read backwards from the current position until you find a label.
> > It's the same algorithm you use to find out what the current
> > section is when you're reading an article.
> 
> No, it doesn't work that way.
> 
> See:
> 
>   foo? (
>     label:
>       dev-foo/bar
>   )
>   dev-bar/foo <-
> 
> Reading backwards, I find 'label'.

...skipping subblocks. Although the case you've given is unlikely to
be used in practice, since experience shows that given this feature,
developers use it in an easily understood manner.

> > > shall we repeat all the disadvantages which you love to forgot to
> > > mention?
> > 
> > No, please stay out of this until you understand the original
> > proposal, and in particular the parts marked with ***. So long as
> > you don't understand what we're discussing, I'm afraid any
> > contributions you make will simply be shouting and waving.
> 
> What have parts marked with '***' to do with the disadvantages we have
> already proved and which you didn't even bother answering?

The problem is that you're arguing against a proposal that doesn't
exist except in your head. If you'd like to read and understand the
proposal being made, which starts with understanding the bits marked
clearly with stars, and then once you've understood it, rethink and
present any issues you find with that proposal then we might have
something to discuss.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 07 Sep 2012 10:50:40 -0400
Ian Stakenvicius <axs@gentoo.org> wrote:
> On 07/09/12 07:45 AM, Ciaran McCreesh wrote:
> > [ Snip! ] Note also how the foo-related things, the bar-related
> > things etc cannot be grouped together by their fooness or barness,
> > but are rather grouped by their DEPENDness and RDEPENDness.
> > 
> > [ Snip! ]
> > 
> > So here's what DEPENDENCIES solves:
> > 
> > Firstly, it allows developers to group together foo-related
> > dependencies and bar-related dependencies by their fooness and
> > barness, not by their role. [ Snip! ] *** It does it by replacing
> > the concept of "a package has build *** dependencies, run
> > dependencies, etc" with "a package has *** dependencies, and each
> > dependency is applicable at one or more of *** build time, run tme,
> > etc".
> 
> And this is the specific point that I don't like about DEPENDENCIES
> versus *DEPEND.  As a developer, I personally find it much more
> straight-forward to fill in the deps needed for each role, rather than
> specifying the role(s) that each dep will play a part in.

Have you tried doing both? You may find you're just arguing from
familiarity, and that after having worked the other way for a few
packages, that the advantages become clearer.

The wide-spread use of hacks like COMMON_DEPEND are a pretty strong
indication that people *do* think in something closer to a
DEPENDENCIES-like fashion. In particular, I find it hard to believe
that you think "ok, so I've got a build dependency upon
>=cat/pkg-2.3[foo]" and then independently work out "ok, I've got a run
>dependency upon >=cat/pkg-2.3[foo]".

- -- 
Ciaran McCreesh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlBKC6cACgkQ96zL6DUtXhG94gCgmXP0C+nAItSnCTIMoeJHzqzK
AyYAniXFZR5mJrBtuGI10dWt0nuJFhwc
=vZBn
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2113 bytes --]

On Fri, 7 Sep 2012 15:53:50 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Fri, 7 Sep 2012 16:23:16 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > On Fri, 7 Sep 2012 13:36:05 +0100
> > Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > > > These are the rules for a machine. People don't actually read
> > > > dependencies sequentially. Provide a good algorithm which works
> > > > from any position.
> > > 
> > > Read backwards from the current position until you find a label.
> > > It's the same algorithm you use to find out what the current
> > > section is when you're reading an article.
> > 
> > No, it doesn't work that way.
> > 
> > See:
> > 
> >   foo? (
> >     label:
> >       dev-foo/bar
> >   )
> >   dev-bar/foo <-
> > 
> > Reading backwards, I find 'label'.
> 
> ...skipping subblocks. Although the case you've given is unlikely to
> be used in practice, since experience shows that given this feature,
> developers use it in an easily understood manner.
> 
> > > > shall we repeat all the disadvantages which you love to forgot
> > > > to mention?
> > > 
> > > No, please stay out of this until you understand the original
> > > proposal, and in particular the parts marked with ***. So long as
> > > you don't understand what we're discussing, I'm afraid any
> > > contributions you make will simply be shouting and waving.
> > 
> > What have parts marked with '***' to do with the disadvantages we
> > have already proved and which you didn't even bother answering?
> 
> The problem is that you're arguing against a proposal that doesn't
> exist except in your head. If you'd like to read and understand the
> proposal being made, which starts with understanding the bits marked
> clearly with stars, and then once you've understood it, rethink and
> present any issues you find with that proposal then we might have
> something to discuss.

Renaming and/or reordering something doesn't change its implications.
It's just 'main disadvantage' vs 'side disadvantage'.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

On Fri, 7 Sep 2012 17:02:57 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> > The problem is that you're arguing against a proposal that doesn't
> > exist except in your head. If you'd like to read and understand the
> > proposal being made, which starts with understanding the bits marked
> > clearly with stars, and then once you've understood it, rethink and
> > present any issues you find with that proposal then we might have
> > something to discuss.
> 
> Renaming and/or reordering something doesn't change its implications.
> It's just 'main disadvantage' vs 'side disadvantage'.

Well no. You've been arguing against some mysterious proposal that
doesn't exist that you think is about replacing lots of variables with
one variable. That's not what DEPENDENCIES is, and now that you've got
a proper write-up, you can try reading it, understanding it and then
seeing if you have any objections to what's actually being proposed.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1241 bytes --]

On Fri, 7 Sep 2012 16:07:54 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Fri, 7 Sep 2012 17:02:57 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > > The problem is that you're arguing against a proposal that doesn't
> > > exist except in your head. If you'd like to read and understand
> > > the proposal being made, which starts with understanding the bits
> > > marked clearly with stars, and then once you've understood it,
> > > rethink and present any issues you find with that proposal then
> > > we might have something to discuss.
> > 
> > Renaming and/or reordering something doesn't change its
> > implications. It's just 'main disadvantage' vs 'side disadvantage'.
> 
> Well no. You've been arguing against some mysterious proposal that
> doesn't exist that you think is about replacing lots of variables with
> one variable. That's not what DEPENDENCIES is, and now that you've got
> a proper write-up, you can try reading it, understanding it and then
> seeing if you have any objections to what's actually being proposed.

Ah, I forgot how the goals change *everything*. Because it's good to
kill hundreds of people for the good reasons.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Wulf C. Krueger]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

(Just for the record, I don't care about the merits or demerits of
*DEPEND or DEPENDENCIES.)

> Ah, I forgot how the goals change *everything*. Because it's good
> to kill hundreds of people for the good reasons.

You might want to take a short break to consider if you're taking
things a bit too seriously or are at least phrasing them a little
strongly here.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlBKEgMACgkQnuVXRcSi+5rfbgCfRyC+mXV1NTDMt+n79q2kQtox
qjwAoMDIcoFcGdYJ9HM2gt9zSDoJp2pC
=RTIv
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Alexis Ballier]

I actually do like the concept but I'm not sure we can reach consensus
about '*DEPEND vs DEPENDENCIES'; a possibility to get people used to it
could be to have two parallel EAPIs, like 6 and 6-dependencies, where
the former will keep the old style and the latter use DEPENDENCIES.

After some time has passed, it could be decided to kill the less useful
one, say in EAPI 8, and get only one 'latest' EAPI again. This decision
doesn't need to be left only to the council, but since it affects
everyone it could be a vote from all the dev community.

There is also the possibility that a consensus will never be reached
and that the two styles will have to live forever, but after all, the
EAPI concept is made for this.

Given this possibility, a debate 'I perfer *DEPEND' vs 'DEPENDENCIES are
more flexible' doesn't make much sense now. What should be discussed is
the concept itself.

For example, what is the HDEPEND equivalent for DEPENDENCIES ? exherbo
documentation doesn't seem to mention an equivalent label.

Alexis.

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1772 bytes --]

On Fri, 7 Sep 2012 12:46:41 -0300
Alexis Ballier <aballier@gentoo.org> wrote:

> I actually do like the concept but I'm not sure we can reach consensus
> about '*DEPEND vs DEPENDENCIES'; a possibility to get people used to
> it could be to have two parallel EAPIs, like 6 and 6-dependencies,
> where the former will keep the old style and the latter use
> DEPENDENCIES.

With eclasses supporting both of them? That's more than crazy.

> After some time has passed, it could be decided to kill the less
> useful one, say in EAPI 8, and get only one 'latest' EAPI again. This
> decision doesn't need to be left only to the council, but since it
> affects everyone it could be a vote from all the dev community.

Why the dev community only? We have many active contributors who aren't
devs and who work hard with ebuilds. It's *their* time which will be
wasted on rewriting dependencies into new form, not yours.

> There is also the possibility that a consensus will never be reached
> and that the two styles will have to live forever, but after all, the
> EAPI concept is made for this.

I believe the correct concept is 'fork'. And that's what Exherbo did.

> Given this possibility, a debate 'I perfer *DEPEND' vs 'DEPENDENCIES
> are more flexible' doesn't make much sense now. What should be
> discussed is the concept itself.
> 
> For example, what is the HDEPEND equivalent for DEPENDENCIES ? exherbo
> documentation doesn't seem to mention an equivalent label.

But Ciaran has just said that this isn't about the labels! It's about
a philosophy in life.

Seriously saying, this isn't about equivalent types. We can choose any
labels we like. It's all about the form in which we will do it.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

On Fri, 7 Sep 2012 12:46:41 -0300
Alexis Ballier <aballier@gentoo.org> wrote:
> For example, what is the HDEPEND equivalent for DEPENDENCIES ? exherbo
> documentation doesn't seem to mention an equivalent label.

DEPENDENCIES is essentially independent of what label names we
introduce. I get the impression Gentoo will be bikeshedding, er, I mean
selecting shorter names for some of the labels than what Exherbo is
using. So HDEPEND could be 'host' if you like.

In any case, the reason you don't see a 'host' label on Exherbo is
because it's called 'build'. Exherbo's taken a more comprehensive
approach to handling ROOT-related dependencies -- dependency resolution
for ROOT!=/ still uses / for satisfying not-purely-runtimeish
dependencies, and then has a way of locking versions on / to versions
in ROOT. It does rely upon having a fully-ROOT-and-/-aware resolver,
though, so it may not be suitable for Gentoo.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/09/12 12:03 PM, Michał Górny wrote:
> On Fri, 7 Sep 2012 12:46:41 -0300 Alexis Ballier
> <aballier@gentoo.org> wrote:
> 
>> I actually do like the concept but I'm not sure we can reach
>> consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
>> people used to it could be to have two parallel EAPIs, like 6 and
>> 6-dependencies, where the former will keep the old style and the
>> latter use DEPENDENCIES.
> 
> With eclasses supporting both of them? That's more than crazy.
> 

By the time EAPI=6* would happen we should have a git tree so we'd
just fork the tree for EAPI=6-dependencies.  :)

A change like this would *NEED* a long-term test phase with a lot of
developer participation, so if we were to consider it we'd need to
fork the tree and implement/maintain it in parallel to main tree
maintenance, imo.  A translation script could probably also be used to
convert *DEPEND into DEPENDENCIES automatically for any packages that
dev's aren't manually managing...

...But I digress; we're nowhere near doing this yet.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBKHK8ACgkQ2ugaI38ACPANNgEAg2K70CzsDPRF7MhBPRoSPtHm
ngLhrWQKqQ7+A74BTJ4BAKxDyUFnAD4ikYrGvo51Ez1FWcGXntQqyiPc/W+491l7
=8+LW
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 869 bytes --]

On 9/7/12 5:46 PM, Alexis Ballier wrote:
> I actually do like the concept but I'm not sure we can reach consensus
> about '*DEPEND vs DEPENDENCIES'

I also like at least significant parts of the DEPENDENCIES concept,
especially when we start adding more dep variables like HDEPEND.

My understanding is that COMMON_DEPEND can be easily and uniformly
expressed as build+run.

Do we have some list of pros and cons, so that the discussion can be
more focused?

> a possibility to get people used to it
> could be to have two parallel EAPIs, like 6 and 6-dependencies, where
> the former will keep the old style and the latter use DEPENDENCIES.

How about just allowing older style DEPEND/RDEPEND, but not both
DEPEND/RDEPEND and DEPENDENCIES in the same ebuild? Note that this may
interact with eclasses (which is also true for above idea I think).


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michael Mol]

On Fri, Sep 7, 2012 at 12:11 PM, Ian Stakenvicius <axs@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 07/09/12 12:03 PM, Michał Górny wrote:
>> On Fri, 7 Sep 2012 12:46:41 -0300 Alexis Ballier
>> <aballier@gentoo.org> wrote:
>>
>>> I actually do like the concept but I'm not sure we can reach
>>> consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
>>> people used to it could be to have two parallel EAPIs, like 6 and
>>> 6-dependencies, where the former will keep the old style and the
>>> latter use DEPENDENCIES.
>>
>> With eclasses supporting both of them? That's more than crazy.
>>
>
> By the time EAPI=6* would happen we should have a git tree so we'd
> just fork the tree for EAPI=6-dependencies.  :)
>
> A change like this would *NEED* a long-term test phase with a lot of
> developer participation, so if we were to consider it we'd need to
> fork the tree and implement/maintain it in parallel to main tree
> maintenance, imo.  A translation script could probably also be used to
> convert *DEPEND into DEPENDENCIES automatically for any packages that
> dev's aren't manually managing...
>
> ...But I digress; we're nowhere near doing this yet.

An intermediate form of that might be useful for auditing the tree and
finding packages which aren't expressing, e.g. RDEPENDS, but probably
should.

-- 
:wq

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

On Fri, 7 Sep 2012 12:28:31 -0400
Michael Mol <mikemol@gmail.com> wrote:
> An intermediate form of that might be useful for auditing the tree and
> finding packages which aren't expressing, e.g. RDEPENDS, but probably
> should.

RDEPEND=DEPEND was removed in EAPI 4, if that's what you mean.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 849 bytes --]

On 9/7/12 6:03 PM, Michał Górny wrote:
> Why the dev community only? We have many active contributors who aren't
> devs and who work hard with ebuilds. It's *their* time which will be
> wasted on rewriting dependencies into new form, not yours.

Should those contributors also vote? Do they have any formal status? Do
we have established procedures for that case?

By the way, for everyone else, there is a forums thread about this topic
here:
<http://forums.gentoo.org/viewtopic-t-935812-start-0-postdays-0-postorder-asc-highlight-.html>

I guess the contributors follow gentoo-dev, so they at least can voice
their concerns, if any.

Also, requiring a rewrite of all existing ebuilds doesn't sound like a
good idea. I think this should be designed not to require a rewrite, and
then the concern about wasted time disappears.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Fri, 07 Sep 2012 18:12:08 +0200
""Paweł Hajdan, Jr."" <phajdan.jr@gentoo.org> wrote:

> On 9/7/12 5:46 PM, Alexis Ballier wrote:
> > I actually do like the concept but I'm not sure we can reach
> > consensus about '*DEPEND vs DEPENDENCIES'
> 
> I also like at least significant parts of the DEPENDENCIES concept,
> especially when we start adding more dep variables like HDEPEND.
> 
> My understanding is that COMMON_DEPEND can be easily and uniformly
> expressed as build+run.
> 
> Do we have some list of pros and cons, so that the discussion can be
> more focused?

There's one on the wiki but I'm not sure if it's up-to-date now. I made
it on top of the previous thread.

http://wiki.gentoo.org/wiki/Future_EAPI/DEPENDENCIES_variable

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 418 bytes --]

On Fri, 07 Sep 2012 18:40:47 +0200
""Paweł Hajdan, Jr."" <phajdan.jr@gentoo.org> wrote:
> Also, requiring a rewrite of all existing ebuilds doesn't sound like a
> good idea. I think this should be designed not to require a rewrite,
> and then the concern about wasted time disappears.

Uh, there is no "rewriting all existing ebuilds" anywhere. I've no idea
where you got that from...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Zac Medico]

On 09/07/2012 09:10 AM, Ciaran McCreesh wrote:
> On Fri, 7 Sep 2012 12:46:41 -0300
> Alexis Ballier <aballier@gentoo.org> wrote:
>> For example, what is the HDEPEND equivalent for DEPENDENCIES ? exherbo
>> documentation doesn't seem to mention an equivalent label.
> 
> DEPENDENCIES is essentially independent of what label names we
> introduce. I get the impression Gentoo will be bikeshedding, er, I mean
> selecting shorter names for some of the labels than what Exherbo is
> using. So HDEPEND could be 'host' if you like.
> 
> In any case, the reason you don't see a 'host' label on Exherbo is
> because it's called 'build'. Exherbo's taken a more comprehensive
> approach to handling ROOT-related dependencies -- dependency resolution
> for ROOT!=/ still uses / for satisfying not-purely-runtimeish
> dependencies, and then has a way of locking versions on / to versions
> in ROOT. It does rely upon having a fully-ROOT-and-/-aware resolver,
> though, so it may not be suitable for Gentoo.

If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you that this is
not a problem. The only reason that Portage currently doesn't have "a
way of locking versions on / to versions in ROOT" is that none of the
existing EAPIs have a way to express this kind of dependency constraint.
-- 
Thanks,
Zacd

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

On Fri, 07 Sep 2012 09:53:46 -0700
Zac Medico <zmedico@gentoo.org> wrote:
> If you're insinuating that Portage may not have a
> "fully-ROOT-and-/-aware resolver", then I can assure you that this is
> not a problem.

In that case, why do we need HDEPEND at all?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/09/12 12:58 PM, Ciaran McCreesh wrote:
> On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico <zmedico@gentoo.org>
> wrote:
>> If you're insinuating that Portage may not have a 
>> "fully-ROOT-and-/-aware resolver", then I can assure you that
>> this is not a problem.
> 
> In that case, why do we need HDEPEND at all?
> 

We don't, actually; HDEPEND is essentially DEPEND.  what we need is
TDEPEND.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBKKKgACgkQ2ugaI38ACPBHFgD/Y7drSHcrEnjNuQZ2bbkLgXPh
6g1KmT4u2xljTR0WergA/jc80lQ2Zy7L3CI2Hn9kyy6cmyZ7yTHJHn5ysdAZkTbS
=kdVT
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Zac Medico]

On 09/07/2012 09:58 AM, Ciaran McCreesh wrote:
> On Fri, 07 Sep 2012 09:53:46 -0700
> Zac Medico <zmedico@gentoo.org> wrote:
>> If you're insinuating that Portage may not have a
>> "fully-ROOT-and-/-aware resolver", then I can assure you that this is
>> not a problem.
> 
> In that case, why do we need HDEPEND at all?

With existing EAPIs, it's possible to use DEPEND for host buildtime-only
deps, RDEPEND for target build+run deps, and PDEPEND for target
runtime-only deps. However, there's no way to to specify buildtime-only
target deps (that aren't needed at runtime). The HDEPEND proposal
involves solves this by using HDEPEND for host buildtime-only deps, and
DEPEND for target buildtime-only deps.
-- 
Thanks,
Zac

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Alexis Ballier]

On Fri, 7 Sep 2012 18:03:51 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> On Fri, 7 Sep 2012 12:46:41 -0300
> Alexis Ballier <aballier@gentoo.org> wrote:
> 
> > I actually do like the concept but I'm not sure we can reach
> > consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
> > people used to it could be to have two parallel EAPIs, like 6 and
> > 6-dependencies, where the former will keep the old style and the
> > latter use DEPENDENCIES.
> 
> With eclasses supporting both of them? That's more than crazy.

depstr=cat/foo

case $EAPI in
 *-dependencies) DEPENDENCIES="build+run: $depstr";;
 *) DEPEND="$depstr"
    RDEPEND="$depstr";;
esac


Yes, eclasses supporting more than one EAPI is crazy and we should
create a new eclass for every EAPI :)


> > After some time has passed, it could be decided to kill the less
> > useful one, say in EAPI 8, and get only one 'latest' EAPI again.
> > This decision doesn't need to be left only to the council, but
> > since it affects everyone it could be a vote from all the dev
> > community.
> 
> Why the dev community only? We have many active contributors who
> aren't devs and who work hard with ebuilds. It's *their* time which
> will be wasted on rewriting dependencies into new form, not yours.

It seems we have a different definition of 'dev community'. That's true
we have well established voting procedures for gentoo devs or foundation
members, but feel free to propose one for the rest of contributors.

> > There is also the possibility that a consensus will never be reached
> > and that the two styles will have to live forever, but after all,
> > the EAPI concept is made for this.
> 
> I believe the correct concept is 'fork'. And that's what Exherbo did.

An EAPI is a fork of the ebuild API already. Exherbo is not a fork as
far as I know, or at least not more than Gentoo is a Redhat fork
because it can process rpm's.

A.

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Zac Medico]

On 09/07/2012 10:02 AM, Ian Stakenvicius wrote:
> On 07/09/12 12:58 PM, Ciaran McCreesh wrote:
>> On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico <zmedico@gentoo.org>
>> wrote:
>>> If you're insinuating that Portage may not have a 
>>> "fully-ROOT-and-/-aware resolver", then I can assure you that
>>> this is not a problem.
> 
>> In that case, why do we need HDEPEND at all?
> 
> 
> We don't, actually; HDEPEND is essentially DEPEND.  what we need is
> TDEPEND.

We could do either one (or do both, and get rid of DEPEND). In
discussions on the chromium-os-dev list [1] (people who could have been
using HDEPEND for years now), the dominant preference was to use HDEPEND
since they felt that it would require the least amount of adjustment to
existing DEPEND settings.

[1]
https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE
-- 
Thanks,
Zac

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/09/12 01:40 PM, Zac Medico wrote:
> On 09/07/2012 10:02 AM, Ian Stakenvicius wrote:
>> On 07/09/12 12:58 PM, Ciaran McCreesh wrote:
>>> On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico
>>> <zmedico@gentoo.org> wrote:
>>>> If you're insinuating that Portage may not have a 
>>>> "fully-ROOT-and-/-aware resolver", then I can assure you
>>>> that this is not a problem.
>> 
>>> In that case, why do we need HDEPEND at all?
>> 
>> 
>> We don't, actually; HDEPEND is essentially DEPEND.  what we need
>> is TDEPEND.
> 
> We could do either one (or do both, and get rid of DEPEND). In 
> discussions on the chromium-os-dev list [1] (people who could have
> been using HDEPEND for years now), the dominant preference was to
> use HDEPEND since they felt that it would require the least amount
> of adjustment to existing DEPEND settings.
> 
> [1] 
> https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE


Thanks
> 
for clarifying this; after reading through the bug I wasn't
sure if the recommendation was to add HDEPEND only or to deprecate
DEPEND entirely for HDEPEND/TDEPEND.

Just to clarify the work involved in converting to this; since DEPEND
on EAPI<=4 is essentially HDEPEND , wouldn't migration to the new EAPI
(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the trivial
ones)?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBKNagACgkQ2ugaI38ACPD7fAD+ItO84yPGTtG5G9aY0nJvTheA
QP4CRV8euHOUeCt1CGsBAK0DbpLXnARHd6lHYCAnuihezRRYr8rO8xw7kIKmlx/U
=DkxI
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/07/2012 10:58 AM, Ian Stakenvicius wrote:
> On 07/09/12 01:40 PM, Zac Medico wrote:
>> On 09/07/2012 10:02 AM, Ian Stakenvicius wrote:
>>> On 07/09/12 12:58 PM, Ciaran McCreesh wrote:
>>>> On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico 
>>>> <zmedico@gentoo.org> wrote:
>>>>> If you're insinuating that Portage may not have a 
>>>>> "fully-ROOT-and-/-aware resolver", then I can assure you 
>>>>> that this is not a problem.
>>> 
>>>> In that case, why do we need HDEPEND at all?
>>> 
>>> 
>>> We don't, actually; HDEPEND is essentially DEPEND.  what we
>>> need is TDEPEND.
> 
>> We could do either one (or do both, and get rid of DEPEND). In 
>> discussions on the chromium-os-dev list [1] (people who could
>> have been using HDEPEND for years now), the dominant preference
>> was to use HDEPEND since they felt that it would require the
>> least amount of adjustment to existing DEPEND settings.
> 
>> [1] 
>> https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE
>
>> 
> 
> Thanks
> 
> for clarifying this; after reading through the bug I wasn't sure if
> the recommendation was to add HDEPEND only or to deprecate DEPEND
> entirely for HDEPEND/TDEPEND.
> 
> Just to clarify the work involved in converting to this; since
> DEPEND on EAPI<=4 is essentially HDEPEND , wouldn't migration to
> the new EAPI (with HDEPEND/DEPEND) generally mean that we would
> need to s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all
> the trivial ones)?

In the linked chromium-os-dev discussion, the consensus seemed to be
that migrating deps from DEPEND to HDEPEND would result in fewer
overall changes than migrating deps from DEPEND to TDEPEND. For this
reason, the dominant preference was to go with HDEPEND.
- -- 
Thanks,
Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlBKOnQACgkQ/ejvha5XGaNguwCg01LvwyvD2AcUG2sTMdTakjuH
P4EAniQQpndputGCSLTcRB2VeZcfFhIj
=5idF
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1002 bytes --]

On Fri, 7 Sep 2012 14:40:25 -0300
Alexis Ballier <aballier@gentoo.org> wrote:

> On Fri, 7 Sep 2012 18:03:51 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > On Fri, 7 Sep 2012 12:46:41 -0300
> > Alexis Ballier <aballier@gentoo.org> wrote:
> > 
> > > I actually do like the concept but I'm not sure we can reach
> > > consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
> > > people used to it could be to have two parallel EAPIs, like 6 and
> > > 6-dependencies, where the former will keep the old style and the
> > > latter use DEPENDENCIES.
> > 
> > With eclasses supporting both of them? That's more than crazy.
> 
> depstr=cat/foo
> 
> case $EAPI in
>  *-dependencies) DEPENDENCIES="build+run: $depstr";;
>  *) DEPEND="$depstr"
>     RDEPEND="$depstr";;
> esac

Yes, we have many eclasses where this is actually the only expected
result. Maybe start with python.eclass, that should be quite an extreme
example.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 07 Sep 2012 11:18:28 -0700
Zac Medico <zmedico@gentoo.org> wrote:
> In the linked chromium-os-dev discussion, the consensus seemed to be
> that migrating deps from DEPEND to HDEPEND would result in fewer
> overall changes than migrating deps from DEPEND to TDEPEND. For this
> reason, the dominant preference was to go with HDEPEND.

They're looking at "minimum number of changes to make it appear to
work", not "minimum number of changes to express dependencies
correctly".

- -- 
Ciaran McCreesh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlBKO44ACgkQ96zL6DUtXhHGDwCgi66mZCVH0UWdB8yfJxl8lcuR
QPkAnRgziWGLpMbJ1i92CXjRJ1Q2tWzJ
=S+jv
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1909 bytes --]

On Fri, 07 Sep 2012 13:58:00 -0400
Ian Stakenvicius <axs@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 07/09/12 01:40 PM, Zac Medico wrote:
> > On 09/07/2012 10:02 AM, Ian Stakenvicius wrote:
> >> On 07/09/12 12:58 PM, Ciaran McCreesh wrote:
> >>> On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico
> >>> <zmedico@gentoo.org> wrote:
> >>>> If you're insinuating that Portage may not have a 
> >>>> "fully-ROOT-and-/-aware resolver", then I can assure you
> >>>> that this is not a problem.
> >> 
> >>> In that case, why do we need HDEPEND at all?
> >> 
> >> 
> >> We don't, actually; HDEPEND is essentially DEPEND.  what we need
> >> is TDEPEND.
> > 
> > We could do either one (or do both, and get rid of DEPEND). In 
> > discussions on the chromium-os-dev list [1] (people who could have
> > been using HDEPEND for years now), the dominant preference was to
> > use HDEPEND since they felt that it would require the least amount
> > of adjustment to existing DEPEND settings.
> > 
> > [1] 
> > https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE
> 
> 
> Thanks
> > 
> for clarifying this; after reading through the bug I wasn't
> sure if the recommendation was to add HDEPEND only or to deprecate
> DEPEND entirely for HDEPEND/TDEPEND.
> 
> Just to clarify the work involved in converting to this; since DEPEND
> on EAPI<=4 is essentially HDEPEND , wouldn't migration to the new EAPI
> (with HDEPEND/DEPEND) generally mean that we would need to
> s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the trivial
> ones)?

I can't agree unless I am missing something. Doesn't the majority of
ebuilds actually require most of DEPEND (well, the part common with
RDEPEND) to be installed on the target? I'm thinking of the shared
libraries mostly.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/07/2012 11:18 AM, Zac Medico wrote:
> On 09/07/2012 10:58 AM, Ian Stakenvicius wrote:
>> On 07/09/12 01:40 PM, Zac Medico wrote:
>>> On 09/07/2012 10:02 AM, Ian Stakenvicius wrote:
>>>> On 07/09/12 12:58 PM, Ciaran McCreesh wrote:
>>>>> On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico 
>>>>> <zmedico@gentoo.org> wrote:
>>>>>> If you're insinuating that Portage may not have a 
>>>>>> "fully-ROOT-and-/-aware resolver", then I can assure you
>>>>>>  that this is not a problem.
>>>> 
>>>>> In that case, why do we need HDEPEND at all?
>>>> 
>>>> 
>>>> We don't, actually; HDEPEND is essentially DEPEND.  what we 
>>>> need is TDEPEND.
> 
>>> We could do either one (or do both, and get rid of DEPEND). In
>>>  discussions on the chromium-os-dev list [1] (people who could 
>>> have been using HDEPEND for years now), the dominant
>>> preference was to use HDEPEND since they felt that it would
>>> require the least amount of adjustment to existing DEPEND
>>> settings.
> 
>>> [1] 
>>> https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE
>
>>> 
>>> 
> 
>> Thanks
> 
>> for clarifying this; after reading through the bug I wasn't sure
>> if the recommendation was to add HDEPEND only or to deprecate
>> DEPEND entirely for HDEPEND/TDEPEND.
> 
>> Just to clarify the work involved in converting to this; since 
>> DEPEND on EAPI<=4 is essentially HDEPEND , wouldn't migration to 
>> the new EAPI (with HDEPEND/DEPEND) generally mean that we would 
>> need to s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie
>> all the trivial ones)?
> 
> In the linked chromium-os-dev discussion, the consensus seemed to
> be that migrating deps from DEPEND to HDEPEND would result in
> fewer overall changes than migrating deps from DEPEND to TDEPEND.
> For this reason, the dominant preference was to go with HDEPEND.

Also, if the HDEPEND proposal is accepted, then I would recommend to
treat the migration of dependencies to HDEPEND as entirely optional.
This is similar to the approach that we took with the introduction of
prefix support in EAPI 3, where EAPI 3 ebuilds were not obligated to
support prefix.
- -- 
Thanks,
Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlBKO6cACgkQ/ejvha5XGaOTVwCfUTyG5yHC6PIoBpsUhWvD874U
bBcAmwfhVLRWRwDBWfI2LvFTdXUY0q7y
=0a/L
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On Fri, 7 Sep 2012 20:23:23 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> I can't agree unless I am missing something. Doesn't the majority of
> ebuilds actually require most of DEPEND (well, the part common with
> RDEPEND) to be installed on the target? I'm thinking of the shared
> libraries mostly.

"The part common with RDEPEND" is a different issue. We're talking
about what the usual thing to do is for dependencies that are in
DEPEND but not RDEPEND. A typical example here is a binary that is
executed as part of the build process.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

On Fri, 7 Sep 2012 19:31:16 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Fri, 7 Sep 2012 20:23:23 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > I can't agree unless I am missing something. Doesn't the majority of
> > ebuilds actually require most of DEPEND (well, the part common with
> > RDEPEND) to be installed on the target? I'm thinking of the shared
> > libraries mostly.
> 
> "The part common with RDEPEND" is a different issue. We're talking
> about what the usual thing to do is for dependencies that are in
> DEPEND but not RDEPEND. A typical example here is a binary that is
> executed as part of the build process.

Thanks for keeping me on the topic.

Now, let me remind you because you probably fail to know the world
outside your dreamworld:

  (with HDEPEND/DEPEND) generally mean that we would need to
  s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the trivial
  ones)?

That does effectively refer to the common depends as well. You know, in
the real world where there is no magical variables which do miracles
behind your back.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

On Fri, 7 Sep 2012 20:46:48 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> Now, let me remind you because you probably fail to know the world
> outside your dreamworld:
> 
>   (with HDEPEND/DEPEND) generally mean that we would need to
>   s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
> trivial ones)?
> 
> That does effectively refer to the common depends as well. You know,
> in the real world where there is no magical variables which do
> miracles behind your back.

Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an
*entirely* different case when it comes to destinations, since RDEPEND
goes to ROOT.

The distinction between DEPEND and HDEPEND is relevant only for
dependencies that are not also in RDEPEND.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

On Fri, 7 Sep 2012 19:52:05 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Fri, 7 Sep 2012 20:46:48 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > Now, let me remind you because you probably fail to know the world
> > outside your dreamworld:
> > 
> >   (with HDEPEND/DEPEND) generally mean that we would need to
> >   s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
> > trivial ones)?
> > 
> > That does effectively refer to the common depends as well. You know,
> > in the real world where there is no magical variables which do
> > miracles behind your back.
> 
> Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an
> *entirely* different case when it comes to destinations, since RDEPEND
> goes to ROOT.

And? RDEPEND isn't required to be merged before building the package.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

On Fri, 7 Sep 2012 21:11:22 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> On Fri, 7 Sep 2012 19:52:05 +0100
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > On Fri, 7 Sep 2012 20:46:48 +0200
> > Michał Górny <mgorny@gentoo.org> wrote:
> > > Now, let me remind you because you probably fail to know the world
> > > outside your dreamworld:
> > > 
> > >   (with HDEPEND/DEPEND) generally mean that we would need to
> > >   s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
> > > trivial ones)?
> > > 
> > > That does effectively refer to the common depends as well. You
> > > know, in the real world where there is no magical variables which
> > > do miracles behind your back.
> > 
> > Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an
> > *entirely* different case when it comes to destinations, since
> > RDEPEND goes to ROOT.
> 
> And? RDEPEND isn't required to be merged before building the package.

I repeat: "things in both DEPEND and RDEPEND".

I also repeat: HDEPEND is relevant only to things that are in DEPEND
and not RDEPEND.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1342 bytes --]

On Fri, 7 Sep 2012 20:13:19 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Fri, 7 Sep 2012 21:11:22 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > On Fri, 7 Sep 2012 19:52:05 +0100
> > Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > > On Fri, 7 Sep 2012 20:46:48 +0200
> > > Michał Górny <mgorny@gentoo.org> wrote:
> > > > Now, let me remind you because you probably fail to know the
> > > > world outside your dreamworld:
> > > > 
> > > >   (with HDEPEND/DEPEND) generally mean that we would need to
> > > >   s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
> > > > trivial ones)?
> > > > 
> > > > That does effectively refer to the common depends as well. You
> > > > know, in the real world where there is no magical variables
> > > > which do miracles behind your back.
> > > 
> > > Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an
> > > *entirely* different case when it comes to destinations, since
> > > RDEPEND goes to ROOT.
> > 
> > And? RDEPEND isn't required to be merged before building the
> > package.
> 
> I repeat: "things in both DEPEND and RDEPEND".
> 
> I also repeat: HDEPEND is relevant only to things that are in DEPEND
> and not RDEPEND.

So... what is your issue in here, sir?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 515 bytes --]

On Fri, 7 Sep 2012 21:21:42 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> So... what is your issue in here, sir?

The issue is what Zac, Ian and I were discussing, before you jumped in
and started yelling. Repeating it for you:

We want to know, for dependencies that are in DEPEND and not RDEPEND,
whether or not most of them will become HDEPENDs, if dependencies are
being expressed properly. If that is the case, then it makes more sense
to introduce TDEPEND than HDEPEND.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/09/12 02:46 PM, Michał Górny wrote:
> On Fri, 7 Sep 2012 19:31:16 +0100 Ciaran McCreesh
> <ciaran.mccreesh@googlemail.com> wrote:
> 
>> On Fri, 7 Sep 2012 20:23:23 +0200 Michał Górny
>> <mgorny@gentoo.org> wrote:
>>> I can't agree unless I am missing something. Doesn't the
>>> majority of ebuilds actually require most of DEPEND (well, the
>>> part common with RDEPEND) to be installed on the target? I'm
>>> thinking of the shared libraries mostly.
>> 
>> "The part common with RDEPEND" is a different issue. We're
>> talking about what the usual thing to do is for dependencies that
>> are in DEPEND but not RDEPEND. A typical example here is a binary
>> that is executed as part of the build process.
> 
> Thanks for keeping me on the topic.
> 
> Now, let me remind you because you probably fail to know the world 
> outside your dreamworld:
> 
> (with HDEPEND/DEPEND) generally mean that we would need to 
> s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
> trivial ones)?
> 
> That does effectively refer to the common depends as well. You
> know, in the real world where there is no magical variables which
> do miracles behind your back.
> 

RDEPEND is defined in an earlier post (by Zac i think?) as target
build+run dependencies; there are very few actual target build-only
dependencies (ie, deps that need to be installed on the target and
don't need to be installed on the host, and don't need to be on the
target after build time).  As such, HDEPEND since it means build-host
build-time deps will still need to have these common dependencies
specified but that doesn't have anything to do with the target's
requirements, which right now are already entirely covered by RDEPEND
(and not DEPEND) at all.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBKTg0ACgkQ2ugaI38ACPBZJwD9E9MT2ZVsKtY+VujuMYjhi5EC
B0RfyRnpXHu4JraBh+0A/014AWaOO3x70axEfdzTFbygRepDe6LWPqFy71nxBC3v
=7hsL
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Alexis Ballier]

On Fri, 7 Sep 2012 20:21:03 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> On Fri, 7 Sep 2012 14:40:25 -0300
> Alexis Ballier <aballier@gentoo.org> wrote:
> 
> > On Fri, 7 Sep 2012 18:03:51 +0200
> > Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > > On Fri, 7 Sep 2012 12:46:41 -0300
> > > Alexis Ballier <aballier@gentoo.org> wrote:
> > > 
> > > > I actually do like the concept but I'm not sure we can reach
> > > > consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
> > > > people used to it could be to have two parallel EAPIs, like 6
> > > > and 6-dependencies, where the former will keep the old style
> > > > and the latter use DEPENDENCIES.
> > > 
> > > With eclasses supporting both of them? That's more than crazy.
> > 
> > depstr=cat/foo
> > 
> > case $EAPI in
> >  *-dependencies) DEPENDENCIES="build+run: $depstr";;
> >  *) DEPEND="$depstr"
> >     RDEPEND="$depstr";;
> > esac
> 
> Yes, we have many eclasses where this is actually the only expected
> result. Maybe start with python.eclass, that should be quite an
> extreme example.
> 

Reference needed. You probably didn't even think more than 2 seconds
before making this claim about python.eclass, because it is not
particularly hard.

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

On Fri, 7 Sep 2012 20:25:58 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Fri, 7 Sep 2012 21:21:42 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> > So... what is your issue in here, sir?
> 
> The issue is what Zac, Ian and I were discussing, before you jumped in
> and started yelling. Repeating it for you:
> 
> We want to know, for dependencies that are in DEPEND and not RDEPEND,
> whether or not most of them will become HDEPENDs, if dependencies are
> being expressed properly. If that is the case, then it makes more
> sense to introduce TDEPEND than HDEPEND.

The only person yelling here is you. I have politely asked a question,
and then you come with your wisdom not answering it at all. And if you
haven't noticed, my question was directed to Ian who -- unlike you --
may be able to say something meaningful in the topic rather than
diverging from it just to prove a random point only you know.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/09/12 02:52 PM, Ciaran McCreesh wrote:
> On Fri, 7 Sep 2012 20:46:48 +0200 Michał Górny <mgorny@gentoo.org>
> wrote:
>> Now, let me remind you because you probably fail to know the
>> world outside your dreamworld:
>> 
>> (with HDEPEND/DEPEND) generally mean that we would need to 
>> s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the 
>> trivial ones)?
>> 
>> That does effectively refer to the common depends as well. You
>> know, in the real world where there is no magical variables which
>> do miracles behind your back.
> 
> Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an 
> *entirely* different case when it comes to destinations, since
> RDEPEND goes to ROOT.
> 
> The distinction between DEPEND and HDEPEND is relevant only for 
> dependencies that are not also in RDEPEND.
> 

Bringing it back to the issue it's solving:

Afaict, for migration:

- - DEPEND changes to HDEPEND
- - the new DEPEND now will be used for things that are *currently* in
RDEPEND and DEPEND (so that things will work) but are not actually
run-time dependencies.  Said atoms will then be removed from RDEPEND
(and not be included in the new HDEPEND either) as they aren't really
supposed to be there in the first place.

Right?

(Note, I have no idea how this will play with PDEPEND but maybe some
of the current circular dependencies will also disappear?)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBKVFUACgkQ2ugaI38ACPCcbwEAr2/+lQ70jPJ0AWp+hGQUo6YG
dFxOJPOoRpBnVlipvnoBAJgQmfB5fqK6lNQB5iJghQYhqRk5KCN4SZe7/lltRviA
=xDo8
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

On Fri, 7 Sep 2012 16:59:48 -0300
Alexis Ballier <aballier@gentoo.org> wrote:

> On Fri, 7 Sep 2012 20:21:03 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > On Fri, 7 Sep 2012 14:40:25 -0300
> > Alexis Ballier <aballier@gentoo.org> wrote:
> > 
> > > On Fri, 7 Sep 2012 18:03:51 +0200
> > > Michał Górny <mgorny@gentoo.org> wrote:
> > > 
> > > > On Fri, 7 Sep 2012 12:46:41 -0300
> > > > Alexis Ballier <aballier@gentoo.org> wrote:
> > > > 
> > > > > I actually do like the concept but I'm not sure we can reach
> > > > > consensus about '*DEPEND vs DEPENDENCIES'; a possibility to
> > > > > get people used to it could be to have two parallel EAPIs,
> > > > > like 6 and 6-dependencies, where the former will keep the old
> > > > > style and the latter use DEPENDENCIES.
> > > > 
> > > > With eclasses supporting both of them? That's more than crazy.
> > > 
> > > depstr=cat/foo
> > > 
> > > case $EAPI in
> > >  *-dependencies) DEPENDENCIES="build+run: $depstr";;
> > >  *) DEPEND="$depstr"
> > >     RDEPEND="$depstr";;
> > > esac
> > 
> > Yes, we have many eclasses where this is actually the only expected
> > result. Maybe start with python.eclass, that should be quite an
> > extreme example.
> > 
> 
> Reference needed. You probably didn't even think more than 2 seconds
> before making this claim about python.eclass, because it is not
> particularly hard.

Hmm, didn't it used to support having python as DEPEND only?

In any case, I'm thinking more of that line. Eclasses which sometimes
add RDEP+DEP, sometimes DEP only, and sometimes do even crazier things.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/09/12 04:10 PM, Michał Górny wrote:
> On Fri, 7 Sep 2012 16:59:48 -0300 Alexis Ballier
> <aballier@gentoo.org> wrote:
> 
>> On Fri, 7 Sep 2012 20:21:03 +0200 Michał Górny
>> <mgorny@gentoo.org> wrote:
>> 
>>> On Fri, 7 Sep 2012 14:40:25 -0300 Alexis Ballier
>>> <aballier@gentoo.org> wrote:
>>> 
>>>> On Fri, 7 Sep 2012 18:03:51 +0200 Michał Górny
>>>> <mgorny@gentoo.org> wrote:
>>>> 
>>>>> On Fri, 7 Sep 2012 12:46:41 -0300 Alexis Ballier
>>>>> <aballier@gentoo.org> wrote:
>>>>> 
>>>>>> I actually do like the concept but I'm not sure we can
>>>>>> reach consensus about '*DEPEND vs DEPENDENCIES'; a
>>>>>> possibility to get people used to it could be to have two
>>>>>> parallel EAPIs, like 6 and 6-dependencies, where the
>>>>>> former will keep the old style and the latter use
>>>>>> DEPENDENCIES.
>>>>> 
>>>>> With eclasses supporting both of them? That's more than
>>>>> crazy.
>>>> 
>>>> depstr=cat/foo
>>>> 
>>>> case $EAPI in *-dependencies) DEPENDENCIES="build+run:
>>>> $depstr";; *) DEPEND="$depstr" RDEPEND="$depstr";; esac
>>> 
>>> Yes, we have many eclasses where this is actually the only
>>> expected result. Maybe start with python.eclass, that should be
>>> quite an extreme example.
>>> 
>> 
>> Reference needed. You probably didn't even think more than 2
>> seconds before making this claim about python.eclass, because it
>> is not particularly hard.
> 
> Hmm, didn't it used to support having python as DEPEND only?
> 
> In any case, I'm thinking more of that line. Eclasses which
> sometimes add RDEP+DEP, sometimes DEP only, and sometimes do even
> crazier things.
> 

Is there anything in particular in the spec/proposal for DEPENDENCIES
that would exclude the addition of individual "build: app-cat/myatom"
"run: app-cat/myatom" deps by an eclass or eclasses?  I know the
"goal" here is to make things atom-centric, but I can't see an
implementation ever working of this that wouldn't permit the "pile-on"
of additional entries of different (or even the same) roles on
identical or near-identical atoms.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBKVZkACgkQ2ugaI38ACPAdAwEAlGthSTR/jor93qpclC5Gl+Sl
82mjHm3ZObOC8Btf+SYA/AxaxCfHuWXoYKj5Ryo9CKna/7cdc1sUoV0fvacO9fja
=AoSy
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 07 Sep 2012 16:08:53 -0400
Ian Stakenvicius <axs@gentoo.org> wrote:
> Bringing it back to the issue it's solving:
> 
> Afaict, for migration:
> 
> - - DEPEND changes to HDEPEND

If we're going by Chromium, AFAICS they're only making this change when
they find they actually need it to get the resolver to give "the right
answer", and otherwise leaving DEPEND as-is. This strikes me as being
heavily in Doing It Wrong territory.

> - - the new DEPEND now will be used for things that are *currently* in
> RDEPEND and DEPEND (so that things will work) but are not actually
> run-time dependencies.  Said atoms will then be removed from RDEPEND
> (and not be included in the new HDEPEND either) as they aren't really
> supposed to be there in the first place.

I'm not entirely sure that there are more than a handful of very
special cases that would be covered by the second point. Can anyone
provide examples?

- -- 
Ciaran McCreesh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlBKVbQACgkQ96zL6DUtXhHcUwCfdNq3MSMyYBAx19ImoOtWFMRM
l2UAoM6DfYJOCL4tLwZ3s6Jeh/6CzBCI
=FIrN
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1506 bytes --]

On Fri, 7 Sep 2012 22:07:30 +0200
Michał Górny <mgorny@gentoo.org> wrote:
> On Fri, 7 Sep 2012 20:25:58 +0100
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > On Fri, 7 Sep 2012 21:21:42 +0200
> > Michał Górny <mgorny@gentoo.org> wrote:
> > > So... what is your issue in here, sir?
> > 
> > The issue is what Zac, Ian and I were discussing, before you jumped
> > in and started yelling. Repeating it for you:
> > 
> > We want to know, for dependencies that are in DEPEND and not
> > RDEPEND, whether or not most of them will become HDEPENDs, if
> > dependencies are being expressed properly. If that is the case,
> > then it makes more sense to introduce TDEPEND than HDEPEND.
> 
> The only person yelling here is you. I have politely asked a question,
> and then you come with your wisdom not answering it at all. And if you
> haven't noticed, my question was directed to Ian who -- unlike you --
> may be able to say something meaningful in the topic rather than
> diverging from it just to prove a random point only you know.

Ian and I are asking essentially the same question regarding the same
issue, and I believe you're the only interested person so far who has
had any difficulty understanding that. However, if hearing it from
someone other than me makes it easier for you to accept that there is
something to discuss, then you're welcome to pretend that everything
that I say from now on was written by someone other than me.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/09/12 04:14 PM, Ciaran McCreesh wrote:
> On Fri, 07 Sep 2012 16:08:53 -0400 Ian Stakenvicius
> <axs@gentoo.org> wrote:
>> Bringing it back to the issue it's solving:
> 
>> Afaict, for migration:
> 
>> - - DEPEND changes to HDEPEND
> 
> If we're going by Chromium, AFAICS they're only making this change
> when they find they actually need it to get the resolver to give
> "the right answer", and otherwise leaving DEPEND as-is. This
> strikes me as being heavily in Doing It Wrong territory.
> 
>> - - the new DEPEND now will be used for things that are
>> *currently* in RDEPEND and DEPEND (so that things will work) but
>> are not actually run-time dependencies.  Said atoms will then be
>> removed from RDEPEND (and not be included in the new HDEPEND
>> either) as they aren't really supposed to be there in the first
>> place.
> 
> I'm not entirely sure that there are more than a handful of very 
> special cases that would be covered by the second point. Can
> anyone provide examples?
> 

Bug 263343 , the 'fontconfig' dep for some package i wasn't able to
find easily

Bug 317337 (original HDEPEND proposal) mentions the
x11-proto/xcb-proto dep for libxcb (and i assume anything else
depending on xcb-proto)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBKWPgACgkQ2ugaI38ACPDUFQEAhOF99mIqr/TXTOGFgdBdARg3
RSPlU8GQpxyEP2x9GPoA/07BSSTfoObc8COCTlpiC+RQP4zbUMwWODazNCszi/hx
=BMvQ
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 07 Sep 2012 16:28:40 -0400
Ian Stakenvicius <axs@gentoo.org> wrote:
> >> - - the new DEPEND now will be used for things that are
> >> *currently* in RDEPEND and DEPEND (so that things will work) but
> >> are not actually run-time dependencies.  Said atoms will then be
> >> removed from RDEPEND (and not be included in the new HDEPEND
> >> either) as they aren't really supposed to be there in the first
> >> place.
> > 
> > I'm not entirely sure that there are more than a handful of very 
> > special cases that would be covered by the second point. Can
> > anyone provide examples?
> 
> Bug 263343 , the 'fontconfig' dep for some package i wasn't able to
> find easily

Do we have an explanation as to *why* fontconfig has to be on ROOT
here? Is it because $ROOT/var/cache/fontconfig needs to exist at build
time, but not at runtime? If so, is this better fixed by using a
temporary directory?

> Bug 317337 (original HDEPEND proposal) mentions the
> x11-proto/xcb-proto dep for libxcb (and i assume anything else
> depending on xcb-proto)

That's a BADEPEND.

- -- 
Ciaran McCreesh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlBKW7cACgkQ96zL6DUtXhG0iQCfdYgzb5kFYE/fN6iTAEJriyuR
zxwAoJi0PvK7JZVqaltliyJFlCZARQRj
=Tzgv
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michael Orlitzky]

On 09/07/2012 07:45 AM, Ciaran McCreesh wrote:
> Since DEPENDENCIES hasn't been written up in a Gentoo-friendly
> manner, and since the Exherbo documentation doesn't seem to suffice
> to explain the idea here, here's some more details on the
> DEPENDENCIES proposal.
> 

It seems to me that the problem this solves is just one of ontology.
It's analogous to trying to stick files named "foo", "bar", "baz",
etc. into directories named "depend", "rdepend", "hdepend", and so on.

There are a few well-known ways to organize things in a hierarchy, and
which one is most efficient depends on the categories and objects that
you have. Given the way that most software is built (see:
COMMON_DEPEND), I think DEPENDENCIES would work better than what we're
doing now, but it also seems more complex.

I think that dependencies are ultimately not hierarchical, and this
can force duplication in DEPENDENCIES as well. Has anyone considered
tagging the package atoms with a list of dependency types? For example,

  * foo/bar: ( build run host )
  * baz/one: baz? ( build )
  * baz/two, baz/three: baz? ( build run )
  ...

This would eliminate duplication of the objects (package atoms) in
favor of duplication of the categories (dependency types). Since the
package atoms are what we really care about, I think the tradeoff is
beneficial. Maintainers get to express each dependency exactly once.

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Patrick Lauer]

On 09/07/12 19:45, Ciaran McCreesh wrote:
> Since DEPENDENCIES hasn't been written up in a Gentoo-friendly manner,
> and since the Exherbo documentation doesn't seem to suffice to explain
> the idea here, here's some more details on the DEPENDENCIES proposal.
>
>
There's change, and there's progress.

This is change, just to have things change.

Like the /usr move, merging udev into some random package no one uses,
and some other things that people do to keep their fingers busy.


Can we please focus on progress instead?

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 294 bytes --]

On Fri, 07 Sep 2012 18:55:10 -0400
Michael Orlitzky <michael@orlitzky.com> wrote:
> I think that dependencies are ultimately not hierarchical

Situations like foo? ( bar? ( || ( a ( b c ) ) ) ) do happen, so any
new syntax would have to be able to deal with that.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]

On Fri, 07 Sep 2012 18:55:10 -0400
Michael Orlitzky <michael@orlitzky.com> wrote:

> On 09/07/2012 07:45 AM, Ciaran McCreesh wrote:
> > Since DEPENDENCIES hasn't been written up in a Gentoo-friendly
> > manner, and since the Exherbo documentation doesn't seem to suffice
> > to explain the idea here, here's some more details on the
> > DEPENDENCIES proposal.
> > 
> 
> It seems to me that the problem this solves is just one of ontology.
> It's analogous to trying to stick files named "foo", "bar", "baz",
> etc. into directories named "depend", "rdepend", "hdepend", and so on.
> 
> There are a few well-known ways to organize things in a hierarchy, and
> which one is most efficient depends on the categories and objects that
> you have. Given the way that most software is built (see:
> COMMON_DEPEND), I think DEPENDENCIES would work better than what we're
> doing now, but it also seems more complex.
> 
> I think that dependencies are ultimately not hierarchical, and this
> can force duplication in DEPENDENCIES as well. Has anyone considered
> tagging the package atoms with a list of dependency types? For
> example,
> 
>   * foo/bar: ( build run host )
>   * baz/one: baz? ( build )
>   * baz/two, baz/three: baz? ( build run )
>   ...
> 
> This would eliminate duplication of the objects (package atoms) in
> favor of duplication of the categories (dependency types). Since the
> package atoms are what we really care about, I think the tradeoff is
> beneficial. Maintainers get to express each dependency exactly once.

This is nowhere near friendly to a developer...

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michael Orlitzky]

On 09/08/2012 02:43 AM, Ciaran McCreesh wrote:
> On Fri, 07 Sep 2012 18:55:10 -0400 Michael Orlitzky
> <michael@orlitzky.com> wrote:
>> I think that dependencies are ultimately not hierarchical
> 
> Situations like foo? ( bar? ( || ( a ( b c ) ) ) ) do happen, so
> any new syntax would have to be able to deal with that.
> 

The deps in both cases are just a collection of atom/type pairs, so
anything possible in one must be possible in the other. I think this
means, if USE=bar, then we need either a or (b and c)? It could be
written,

|| (
  a: bar? ( build run )
  b,c: bar? ( build run )
)

Or if we wanted to make it even easier, allow the USE conditional at
the top level like we do now:

bar? ( || (
  a: ( build run )
  b,c: ( build run )
))

I'm just wondering if it wouldn't be nicer to think in terms of
package atoms instead of the dependency types. Right now, we've got
buckets named DEPEND, RDEPEND, etc. and we put the package atoms in
those buckets. The above syntax would make the package atoms the
buckets, and we would be putting the dependency types into the buckets
instead.

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Matt Turner]

On Fri, Sep 7, 2012 at 4:45 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> Since DEPENDENCIES hasn't been written up in a Gentoo-friendly manner,
> and since the Exherbo documentation doesn't seem to suffice to explain
> the idea here, here's some more details on the DEPENDENCIES proposal.

I like this. It seems well planned and thought out and very flexible.
If I understand correctly paludis already supports this, so a working
implementation is already available which is clearly beneficial for a
proposal.

Thanks a lot, Ciaran, for writing this up.

Matt

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Brian Harring]

On Fri, Sep 07, 2012 at 04:14:17PM -0400, Ian Stakenvicius wrote:
> Is there anything in particular in the spec/proposal for DEPENDENCIES
> that would exclude the addition of individual "build: app-cat/myatom"
> "run: app-cat/myatom" deps by an eclass or eclasses?  I know the
> "goal" here is to make things atom-centric, but I can't see an
> implementation ever working of this that wouldn't permit the "pile-on"
> of additional entries of different (or even the same) roles on
> identical or near-identical atoms.

They could be piled on; it would require each eclass to reset the 
label for safety reasons though; same goes for ebuilds frankly (or the 
PM would have to reset the context to build+run: each time through).

Pardon if addressed elsewhere; this thread is a fucking mess...
~harring

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Kent Fredric]

On 11 September 2012 14:16, Brian Harring <ferringb@gmail.com> wrote:
> On Fri, Sep 07, 2012 at 04:14:17PM -0400, Ian Stakenvicius wrote:
>> Is there anything in particular in the spec/proposal for DEPENDENCIES
>> that would exclude the addition of individual "build: app-cat/myatom"
>> "run: app-cat/myatom" deps by an eclass or eclasses?  I know the
>> "goal" here is to make things atom-centric, but I can't see an
>> implementation ever working of this that wouldn't permit the "pile-on"
>> of additional entries of different (or even the same) roles on
>> identical or near-identical atoms.
>
> They could be piled on; it would require each eclass to reset the
> label for safety reasons though; same goes for ebuilds frankly (or the
> PM would have to reset the context to build+run: each time through).
>
> Pardon if addressed elsewhere; this thread is a fucking mess...
> ~harring
>
Correct me if I'm wrong, but couldn't the entire proposition could be
implemented in an eclass, not needing the EAPI development cycle to be
tied up with it.

All you need is something in bash that can parse DEPENDENCIES and
populate *DEPEND , and the underlying guts could be done in
practically any language without requiring PM specific
implementations.

[[[

inherit polydep;

DEPENDENCIES="
   Stuff Here.
";

]]]

The only thing I know of that is limiting the above from being
implemented that way is the lack of post-source eclass hooks, that is:
 currently you'd have to do either

[[[

DEPENDENCIES="..."
inherit polydep;

]]]

or use a callback

[[[

inherit polydep;
DEPENDENCIES=" ... "

polydeps;

]]]


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Brian Harring]

On Fri, Sep 14, 2012 at 07:18:54AM +1200, Kent Fredric wrote:
> On 11 September 2012 14:16, Brian Harring <ferringb@gmail.com> wrote:
> > On Fri, Sep 07, 2012 at 04:14:17PM -0400, Ian Stakenvicius wrote:
> >> Is there anything in particular in the spec/proposal for DEPENDENCIES
> >> that would exclude the addition of individual "build: app-cat/myatom"
> >> "run: app-cat/myatom" deps by an eclass or eclasses?  I know the
> >> "goal" here is to make things atom-centric, but I can't see an
> >> implementation ever working of this that wouldn't permit the "pile-on"
> >> of additional entries of different (or even the same) roles on
> >> identical or near-identical atoms.
> >
> > They could be piled on; it would require each eclass to reset the
> > label for safety reasons though; same goes for ebuilds frankly (or the
> > PM would have to reset the context to build+run: each time through).
> >
> > Pardon if addressed elsewhere; this thread is a fucking mess...
> > ~harring
> >
> Correct me if I'm wrong, but couldn't the entire proposition could be
> implemented in an eclass, not needing the EAPI development cycle to be
> tied up with it.
> 
> All you need is something in bash that can parse DEPENDENCIES and
> populate *DEPEND , and the underlying guts could be done in
> practically any language without requiring PM specific
> implementations.

You've got it inverted; if any autopopulation is occuring, *DEPEND -> 
DEPENDENCIES is the sane form.

While it definitely *is* possible to render DEPENDENCIES down into 
depend/rdepend (after all, the PM has to do exactly this for 
resolution), that does /not/ mean doing it in bash is a good idea.

I'd really not want to try that using labels; using use conditionals 
('dep:run,build? ( targets )') is frankly a bit easier imo, but still; 
why do so unless one likes pain?  It doesn't actually gain us 
anything via missing the point of DEPENDENCIES.

The point of unified DEPENDENCIES var (regardless of the form) is 
thus:
1) ability to specify common deps once, w/out having to use 
intermediate vars/copy-pasting/etc.  Think COMMON_DEPEND, and this 
should make sense.

2) To shift to a form where adding new dependency targets is easy- 
whether it be sdepend, fdepend, tdepend, or hdepend (or 
ONE-RING-DEPEND to rule them all).  This actually is rather important; 
for the average 95% case, devs won't actually have to pay much 
attention to those vars; but for those of us a bit further out (cross 
compilation, heavy parallelization, etc) those depend forms are 
becoming increasingly painful in their absense.


Basically, having devs specify DEPENDENCIES in ebuilds, which then an 
eclass chunks out into DEPEND/RDEPEND misses the point of this; it's 
doable, it's just not particularly sane imo.

The other way around, having *DEPEND automatically be collapsed into 
DEPENDENCIES, however is very sane- it makes transition/compatibilty 
for devs bloody simple, while structuring it so we can do further 
enhancements.

~harring

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Kent Fredric]

On 14 September 2012 10:17, Brian Harring <ferringb@gmail.com> wrote:
>> All you need is something in bash that can parse DEPENDENCIES and
>> populate *DEPEND , and the underlying guts could be done in
>> practically any language without requiring PM specific
>> implementations.
>
> You've got it inverted; if any autopopulation is occuring, *DEPEND ->
> DEPENDENCIES is the sane form.
>
> While it definitely *is* possible to render DEPENDENCIES down into
> depend/rdepend (after all, the PM has to do exactly this for
> resolution), that does /not/ mean doing it in bash is a good idea.
>
> I'd really not want to try that using labels; using use conditionals
> ('dep:run,build? ( targets )') is frankly a bit easier imo, but still;
> why do so unless one likes pain?  It doesn't actually gain us
> anything via missing the point of DEPENDENCIES.
>
> The point of unified DEPENDENCIES var (regardless of the form) is
> thus:
> 1) ability to specify common deps once, w/out having to use
> intermediate vars/copy-pasting/etc.  Think COMMON_DEPEND, and this
> should make sense.
>
> 2) To shift to a form where adding new dependency targets is easy-
> whether it be sdepend, fdepend, tdepend, or hdepend (or
> ONE-RING-DEPEND to rule them all).  This actually is rather important;
> for the average 95% case, devs won't actually have to pay much
> attention to those vars; but for those of us a bit further out (cross
> compilation, heavy parallelization, etc) those depend forms are
> becoming increasingly painful in their absense.
>
>
> Basically, having devs specify DEPENDENCIES in ebuilds, which then an
> eclass chunks out into DEPEND/RDEPEND misses the point of this; it's
> doable, it's just not particularly sane imo.
>
> The other way around, having *DEPEND automatically be collapsed into
> DEPENDENCIES, however is very sane- it makes transition/compatibilty
> for devs bloody simple, while structuring it so we can do further
> enhancements.
>
> ~harring
>


Sure, but at least this makes it a viable proof-of-concept without
needing all the different PM's to implement the new spec first, and
due to not being EAPI bound when done this way, means you can just do
it and have it work both now and in the future.

And because of this "experimental" nature, you don't have to do *ALL*
the parsing in bash, you could make the eclass use some external code
to parse it and spit it out, and simply have the eclass depend on that
external program regardless.

I agree that long term, a Unified DEPENDENCIES implementation is the
way forward, but if you want to convince people, having something
which has been demonstrated and tested in a real world setting goes a
long way.

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Brian Harring]

On Sat, Sep 15, 2012 at 11:06:01PM +1200, Kent Fredric wrote:
> On 14 September 2012 10:17, Brian Harring <ferringb@gmail.com> wrote:
> >> All you need is something in bash that can parse DEPENDENCIES and
> >> populate *DEPEND , and the underlying guts could be done in
> >> practically any language without requiring PM specific
> >> implementations.
> >
> > You've got it inverted; if any autopopulation is occuring, *DEPEND ->
> > DEPENDENCIES is the sane form.
> >
> > While it definitely *is* possible to render DEPENDENCIES down into
> > depend/rdepend (after all, the PM has to do exactly this for
> > resolution), that does /not/ mean doing it in bash is a good idea.
> >
> > I'd really not want to try that using labels; using use conditionals
> > ('dep:run,build? ( targets )') is frankly a bit easier imo, but still;
> > why do so unless one likes pain?  It doesn't actually gain us
> > anything via missing the point of DEPENDENCIES.
> >
> > The point of unified DEPENDENCIES var (regardless of the form) is
> > thus:
> > 1) ability to specify common deps once, w/out having to use
> > intermediate vars/copy-pasting/etc.  Think COMMON_DEPEND, and this
> > should make sense.
> >
> > 2) To shift to a form where adding new dependency targets is easy-
> > whether it be sdepend, fdepend, tdepend, or hdepend (or
> > ONE-RING-DEPEND to rule them all).  This actually is rather important;
> > for the average 95% case, devs won't actually have to pay much
> > attention to those vars; but for those of us a bit further out (cross
> > compilation, heavy parallelization, etc) those depend forms are
> > becoming increasingly painful in their absense.
> >
> >
> > Basically, having devs specify DEPENDENCIES in ebuilds, which then an
> > eclass chunks out into DEPEND/RDEPEND misses the point of this; it's
> > doable, it's just not particularly sane imo.
> >
> > The other way around, having *DEPEND automatically be collapsed into
> > DEPENDENCIES, however is very sane- it makes transition/compatibilty
> > for devs bloody simple, while structuring it so we can do further
> > enhancements.
> >
> > ~harring
> >
> 
> 
> Sure, but at least this makes it a viable proof-of-concept without
> needing all the different PM's to implement the new spec first, and
> due to not being EAPI bound when done this way, means you can just do
> it and have it work both now and in the future.
>
> And because of this "experimental" nature, you don't have to do *ALL*
> the parsing in bash, you could make the eclass use some external code
> to parse it and spit it out, and simply have the eclass depend on that
> external program regardless.
>
> I agree that long term, a Unified DEPENDENCIES implementation is the
> way forward, but if you want to convince people, having something
> which has been demonstrated and tested in a real world setting goes a
> long way.

Honestly, QA would be well within their rights to kick anyone who did 
this, *hard* in the shins.

I understand your notion- specifically proof of concept, show the 
data, etc; I just think you've still got inverted, too focused on 
trying to do it in bash.

To demonstrate the gain of this, we basically take the existing tree's 
deps, and re-render it into a unified DEPENDENCIES form.

As for adding support to a PM, if we use the use conditional proposal 
of mine, it's bloody simple- the PM already supports it, we just need 
some minor USE_EXPAND adjustments.

~harring

Re: [gentoo-dev] Unified DEPENDENCIES concept

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3860 bytes --]

On Sat, 15 Sep 2012 13:33:18 -0700
Brian Harring <ferringb@gmail.com> wrote:

> On Sat, Sep 15, 2012 at 11:06:01PM +1200, Kent Fredric wrote:
> > On 14 September 2012 10:17, Brian Harring <ferringb@gmail.com>
> > wrote:
> > >> All you need is something in bash that can parse DEPENDENCIES and
> > >> populate *DEPEND , and the underlying guts could be done in
> > >> practically any language without requiring PM specific
> > >> implementations.
> > >
> > > You've got it inverted; if any autopopulation is occuring,
> > > *DEPEND -> DEPENDENCIES is the sane form.
> > >
> > > While it definitely *is* possible to render DEPENDENCIES down into
> > > depend/rdepend (after all, the PM has to do exactly this for
> > > resolution), that does /not/ mean doing it in bash is a good idea.
> > >
> > > I'd really not want to try that using labels; using use
> > > conditionals ('dep:run,build? ( targets )') is frankly a bit
> > > easier imo, but still; why do so unless one likes pain?  It
> > > doesn't actually gain us anything via missing the point of
> > > DEPENDENCIES.
> > >
> > > The point of unified DEPENDENCIES var (regardless of the form) is
> > > thus:
> > > 1) ability to specify common deps once, w/out having to use
> > > intermediate vars/copy-pasting/etc.  Think COMMON_DEPEND, and this
> > > should make sense.
> > >
> > > 2) To shift to a form where adding new dependency targets is easy-
> > > whether it be sdepend, fdepend, tdepend, or hdepend (or
> > > ONE-RING-DEPEND to rule them all).  This actually is rather
> > > important; for the average 95% case, devs won't actually have to
> > > pay much attention to those vars; but for those of us a bit
> > > further out (cross compilation, heavy parallelization, etc) those
> > > depend forms are becoming increasingly painful in their absense.
> > >
> > >
> > > Basically, having devs specify DEPENDENCIES in ebuilds, which
> > > then an eclass chunks out into DEPEND/RDEPEND misses the point of
> > > this; it's doable, it's just not particularly sane imo.
> > >
> > > The other way around, having *DEPEND automatically be collapsed
> > > into DEPENDENCIES, however is very sane- it makes
> > > transition/compatibilty for devs bloody simple, while structuring
> > > it so we can do further enhancements.
> > >
> > > ~harring
> > >
> > 
> > 
> > Sure, but at least this makes it a viable proof-of-concept without
> > needing all the different PM's to implement the new spec first, and
> > due to not being EAPI bound when done this way, means you can just
> > do it and have it work both now and in the future.
> >
> > And because of this "experimental" nature, you don't have to do
> > *ALL* the parsing in bash, you could make the eclass use some
> > external code to parse it and spit it out, and simply have the
> > eclass depend on that external program regardless.
> >
> > I agree that long term, a Unified DEPENDENCIES implementation is the
> > way forward, but if you want to convince people, having something
> > which has been demonstrated and tested in a real world setting goes
> > a long way.
> 
> Honestly, QA would be well within their rights to kick anyone who did 
> this, *hard* in the shins.
> 
> I understand your notion- specifically proof of concept, show the 
> data, etc; I just think you've still got inverted, too focused on 
> trying to do it in bash.
> 
> To demonstrate the gain of this, we basically take the existing
> tree's deps, and re-render it into a unified DEPENDENCIES form.

But in order to do this, we first have to decide exactly what kind
of dependencies do we want to have. Then convert the tree to
a separate-variable form with new dependencies. Then we can compare
it with the DEPENDENCIES form and decide which one is better.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

[gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Brian Harring]

On Sun, Sep 16, 2012 at 12:03:36AM +0200, Micha?? G??rny wrote:
> On Sat, 15 Sep 2012 13:33:18 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> > To demonstrate the gain of this, we basically take the existing
> > tree's deps, and re-render it into a unified DEPENDENCIES form.
> 
> But in order to do this, we first have to decide exactly what kind
> of dependencies do we want to have. Then convert the tree to
> a separate-variable form with new dependencies. Then we can compare
> it with the DEPENDENCIES form and decide which one is better.

Funny you mentioned that, I just finished tweaking pquery to generate 
real world example unified dependencies; these *are* accurate, just to 
be clear.

Dumps are at 
http://dev.gentoo.org/~ferringb/unified-dependencies-example/ .

Herds, if you want to see what your pkgs would look like, look at 
http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ .

If you'd like to see an *example effect* it has on what gets displayed 
to the user (aka, after all major use conditionals are stripped), look 
at 
http://dev.gentoo.org/~ferringb/unified-dependencies-example/user-visible.txt 
; warning, that's a 55MB file.  The syntax in use there isn't great, 
but as said, it's an example.

Total cache savings from doing this for a full tree conversion, for 
our existing md5-cache format is 2.73MB (90 byes per cache entry).  
Calculating the savings from the ebuild/eclass standpoint is dependent 
on how the deps are built up, so I skipped that.

The algorithim used is fairly stupid, but reasonably effectively; 
essentially it intersects the top level of each individual type of 
dep, breaking out common groupings.

In other words, it won't pick up this:
DEPEND="x? ( dev-util/diffball dev-util/bsdiff )"
RDEPEND="x? ( dev-util/diffball )"

and convert it into thus
DEPENDENCIES="
dep:build,run? (
  x? ( 
       dev-util/diffball
       dep:run? ( 
         dev-util/diffball
       )
     )
  )"

Additionally, the form used here makes *no assumption about default 
context*; in any final solution we use, a default context would be 
wise- say build,run.  Again, an example of what I mean.

If we said "in the absense of a context, the default is dep:build,run" 
the following:

DEPEND="dev-util/diffball dev-util/bsdiff"
RDEPEND="dev-util/diffball de-vutil/bsdiff x? ( sys-apps/pkgcore )"
PDEPEND="dev-python/snakeoil"

would be:
DEPENDENCIES="
  dev-util/diffball
  dev-util/bsdiff
  dep:run? ( x? ( sys-apps/pkgcore ) )
  dep:post? ( dev-python/snakeoil )
"

The quicky algo I used assumes no default context, thus it writes 
this:
DEPENDENCIES="
  dep:build,run? (
    dev-util/diffball
    dev-util/bsdiff
  )
  dep:run? ( x? ( sys-apps/pkgcore ) )
  dep:post? ( dev-python/snakeoil )
"

Etc.

~harring

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Diego Elio Pettenò]

On 15/09/2012 18:20, Brian Harring wrote:
> Herds, if you want to see what your pkgs would look like, look at 
> http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ .

Ruby team could make use of a dep:test and automatic conversion of that :P

-- 
Diego Elio Pettenò — Flameeyes
flameeyes@flameeyes.eu — http://blog.flameeyes.eu/

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Ben de Groot]

On 16 September 2012 09:20, Brian Harring <ferringb@gmail.com> wrote:
> On Sun, Sep 16, 2012 at 12:03:36AM +0200, Micha?? G??rny wrote:
>> On Sat, 15 Sep 2012 13:33:18 -0700
>> Brian Harring <ferringb@gmail.com> wrote:
>> > To demonstrate the gain of this, we basically take the existing
>> > tree's deps, and re-render it into a unified DEPENDENCIES form.
>>
>> But in order to do this, we first have to decide exactly what kind
>> of dependencies do we want to have. Then convert the tree to
>> a separate-variable form with new dependencies. Then we can compare
>> it with the DEPENDENCIES form and decide which one is better.
>
> Funny you mentioned that, I just finished tweaking pquery to generate
> real world example unified dependencies; these *are* accurate, just to
> be clear.
>
> Dumps are at
> http://dev.gentoo.org/~ferringb/unified-dependencies-example/ .
>
> Herds, if you want to see what your pkgs would look like, look at
> http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ .
>
> If you'd like to see an *example effect* it has on what gets displayed
> to the user (aka, after all major use conditionals are stripped), look
> at
> http://dev.gentoo.org/~ferringb/unified-dependencies-example/user-visible.txt
> ; warning, that's a 55MB file.  The syntax in use there isn't great,
> but as said, it's an example.
>
> Total cache savings from doing this for a full tree conversion, for
> our existing md5-cache format is 2.73MB (90 byes per cache entry).
> Calculating the savings from the ebuild/eclass standpoint is dependent
> on how the deps are built up, so I skipped that.
>
> The algorithim used is fairly stupid, but reasonably effectively;
> essentially it intersects the top level of each individual type of
> dep, breaking out common groupings.
>
> In other words, it won't pick up this:
> DEPEND="x? ( dev-util/diffball dev-util/bsdiff )"
> RDEPEND="x? ( dev-util/diffball )"
>
> and convert it into thus
> DEPENDENCIES="
> dep:build,run? (
>   x? (
>        dev-util/diffball
>        dep:run? (
>          dev-util/diffball
>        )
>      )
>   )"
>
> Additionally, the form used here makes *no assumption about default
> context*; in any final solution we use, a default context would be
> wise- say build,run.  Again, an example of what I mean.
>
> If we said "in the absense of a context, the default is dep:build,run"
> the following:
>
> DEPEND="dev-util/diffball dev-util/bsdiff"
> RDEPEND="dev-util/diffball de-vutil/bsdiff x? ( sys-apps/pkgcore )"
> PDEPEND="dev-python/snakeoil"
>
> would be:
> DEPENDENCIES="
>   dev-util/diffball
>   dev-util/bsdiff
>   dep:run? ( x? ( sys-apps/pkgcore ) )
>   dep:post? ( dev-python/snakeoil )
> "
>
> The quicky algo I used assumes no default context, thus it writes
> this:
> DEPENDENCIES="
>   dep:build,run? (
>     dev-util/diffball
>     dev-util/bsdiff
>   )
>   dep:run? ( x? ( sys-apps/pkgcore ) )
>   dep:post? ( dev-python/snakeoil )
> "
>
> Etc.
>
> ~harring
>

Thanks. I have given it a quick overview for the qt herd. I still
don't see what using DEPENDENCIES adds to what we do now with separate
*DEPEND variables. I see no convincing reason to change what we do.

As I've said before on IRC, we need a good costs/benefits overview.
Right now I only see costs (migrating ebuilds and eclasses) and no
benefits.

-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1390 bytes --]

On Sat, 15 Sep 2012 18:20:26 -0700
Brian Harring <ferringb@gmail.com> wrote:

> On Sun, Sep 16, 2012 at 12:03:36AM +0200, Micha?? G??rny wrote:
> > On Sat, 15 Sep 2012 13:33:18 -0700
> > Brian Harring <ferringb@gmail.com> wrote:
> > > To demonstrate the gain of this, we basically take the existing
> > > tree's deps, and re-render it into a unified DEPENDENCIES form.
> > 
> > But in order to do this, we first have to decide exactly what kind
> > of dependencies do we want to have. Then convert the tree to
> > a separate-variable form with new dependencies. Then we can compare
> > it with the DEPENDENCIES form and decide which one is better.
> 
> Funny you mentioned that, I just finished tweaking pquery to generate 
> real world example unified dependencies; these *are* accurate, just
> to be clear.

But consider that for example Zac & AxS (correct me if I recall it
correctly) considered making changing the meaning of RDEPEND to install
them before the build, thus effectively making 'build,run' useless.

> Total cache savings from doing this for a full tree conversion, for 
> our existing md5-cache format is 2.73MB (90 byes per cache entry).  
> Calculating the savings from the ebuild/eclass standpoint is
> dependent on how the deps are built up, so I skipped that.

You're storing the cache in a tarball?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Brian Harring]

On Sun, Sep 16, 2012 at 09:56:27AM +0200, Micha?? G??rny wrote:
> But consider that for example Zac & AxS (correct me if I recall it
> correctly) considered making changing the meaning of RDEPEND to install
> them before the build, thus effectively making 'build,run' useless.

I really am not trying to be a blatant dick to you, but this has 
/zero/ relevance.  RDEPEND means "required for runtime".  That ain't 
changing.  If they were discussing changing what RDEPEND meant, then 
they were high, period.

If zac/axs want to try and make the resolver install RDEPEND before 
DEPEND... well, they're free to.  That doesn't change the fact that 
the deps still must be specified correctly; in short, build,run is 
very much relevant.

What I suspect they were intending on doing is letting the resolver 
work on RDEPENDS of a pkg in parallel to that pkg being built; this is 
a parallelization scheduling optimization, still requires accurate 
deps.

I'm trying to be nice here, but you're very confused on this matter.


> > Total cache savings from doing this for a full tree conversion, for 
> > our existing md5-cache format is 2.73MB (90 byes per cache entry).  
> > Calculating the savings from the ebuild/eclass standpoint is
> > dependent on how the deps are built up, so I skipped that.
> 
> You're storing the cache in a tarball?

Going to assume you're not trolling, and instead use this as a 
way to point out that this actually *does* matter, although it's 
admittedly not obvious if you don't know much about the guts of 
package managers, or don't spend your saturday nights doing fun 
things like optimizing ebuild package manager performance.

First, the figure is 3.204MB if default context is used; ~9.5% of the 
content footprint for md5-cache specifically.

Little known fact; rsync transfers for gentoo are required to be 
--whole-file; meaning no intra-file delta compression, it transfers 
the whole file itself.  This is done to keep cpu load on rsync nodes 
low (else they'd be calculating minimally 97k md4's for every sync, 
not counting the rolling adl32 chksum for all content dependent on 
the window cut off threshold- sounds minor, but it's death by a 
thousand cuts).

For obvious reasons, the cache is the hottest part of the tree due to 
cascading updates due to eclass changes.  In other words, that ~9.5%
reduction targest the core data actually transferered in a sync.

In terms of the total tree footprint, it's a 1% reduction; mostly lost 
in blocksize overhead unless you're using squashfs (which a decent 
number of folks do for speed reasons), or use tail packing FS for the 
tree (again, more than you'd think- known primarily due to reiserfs 
corruption bugs causing some hell on PM caches).

There's also the fact doing this means best case, 2 less inodes per 
VDB entry (more once we start adding dependency types).  For my vdb, I 
have 15523 across 798 pkgs.  1331 of that is *DEPEND, converted to 
DEPENDENCIES the file count is 748.  Note that's preserving DEPEND, 
although it's worthless at this stage of the vdb.  So 5% reduction in 
files in there.  Whoopy-de-doo, right?

This one I can't test as well since the only rotational media I've got 
these days is a hardware raid w/ a beefy cache; the closest I can 
manage is local network nfs to an ssd FS, so it'll have to serve 
as a stand in for cold cache/hot cache, and for a demonstration of 
why having a backend that is a 101 small individual files is bad.

Best of 5 is displayed below:

Iterating over the vdb, and parsing and rendering all depends for our 
current layout, w/ the vdb stored on nfs:

cold cache:
real    0m30.405s
user    0m1.046s
sys     0m0.390s

hot cache:
real    0m16.483s
user    0m0.883s
sys     0m0.168s

non-optimized, hacked to work (known slower for parsing in comparison 
to the non quicky hack), iterating over the vdb, parsing all
depends and rendering said depends when it's stored as DEPENDENCIES; 
literally, rendering DEPEND from it, RDEPEND, PDEPEND.

cold cache:
real    0m18.329s
user    0m0.908s
sys     0m0.280s

hot cache
real    0m12.185s
user    0m0.860s
sys     0m0.128s


You get the idea.  See the various infamous cold cache/hot cache 
performance tests in doubt; I can tell you that a similar trick, done 
in '07, literally just skipping loading USE till it was needed for 
provides parsing was enough to bring a 5400RPM drive's run time 
down from 15s to 12s for cold cache- for parsing provides *alone*, 
nothing else.  Either way, do your own investigation, it's a
good education on performance.


Hopefully for the others listening, that last section was a random but 
useful tidbit of info; if not, pardon, just being through to make sure 
this point is not raised again.

~harring

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1685 bytes --]

On Sun, 16 Sep 2012 04:10:01 -0700
Brian Harring <ferringb@gmail.com> wrote:

> On Sun, Sep 16, 2012 at 09:56:27AM +0200, Micha?? G??rny wrote:
> > But consider that for example Zac & AxS (correct me if I recall it
> > correctly) considered making changing the meaning of RDEPEND to
> > install them before the build, thus effectively making 'build,run'
> > useless.
> 
> I really am not trying to be a blatant dick to you, but this has 
> /zero/ relevance.  RDEPEND means "required for runtime".  That ain't 
> changing.  If they were discussing changing what RDEPEND meant, then 
> they were high, period.
> 
> If zac/axs want to try and make the resolver install RDEPEND before 
> DEPEND... well, they're free to.  That doesn't change the fact that 
> the deps still must be specified correctly; in short, build,run is 
> very much relevant.

I don't think we have made up our mind what *exactly* we want from
deps. Just because we have something semi-correct right now, doesn't
mean that we don't want to change that. But I guess with the whole
amount of noise in here I won't ever get any definitive answer.

> There's also the fact doing this means best case, 2 less inodes per 
> VDB entry (more once we start adding dependency types).  For my vdb,
> I have 15523 across 798 pkgs.  1331 of that is *DEPEND, converted to 
> DEPENDENCIES the file count is 748.  Note that's preserving DEPEND, 
> although it's worthless at this stage of the vdb.  So 5% reduction in 
> files in there.  Whoopy-de-doo, right?

So we can modify vdb now? What about all those applications which
obviously are broken due to that?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Brian Harring]

On Sun, Sep 16, 2012 at 01:21:26PM +0200, Micha?? G??rny wrote:
> On Sun, 16 Sep 2012 04:10:01 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> 
> > On Sun, Sep 16, 2012 at 09:56:27AM +0200, Micha?? G??rny wrote:
> > > But consider that for example Zac & AxS (correct me if I recall it
> > > correctly) considered making changing the meaning of RDEPEND to
> > > install them before the build, thus effectively making 'build,run'
> > > useless.
> > 
> > I really am not trying to be a blatant dick to you, but this has 
> > /zero/ relevance.  RDEPEND means "required for runtime".  That ain't 
> > changing.  If they were discussing changing what RDEPEND meant, then 
> > they were high, period.
> > 
> > If zac/axs want to try and make the resolver install RDEPEND before 
> > DEPEND... well, they're free to.  That doesn't change the fact that 
> > the deps still must be specified correctly; in short, build,run is 
> > very much relevant.
> 
> I don't think we have made up our mind what *exactly* we want from
> deps.

Are we now expecting deps to give us ponies or something?  We know 
*exactly* what we want from deps, and their current definition- the 
problem isn't the definition, it's that we don't have the forms we 
need.


> Just because we have something semi-correct right now, doesn't
> mean that we don't want to change that.

This is a no-op argument against the proposal: "we can't 
change the deps because we might want to change the deps".  It's also 
irrelevant due to the core basis of it being broken as fuck (described 
above).


> > There's also the fact doing this means best case, 2 less inodes per 
> > VDB entry (more once we start adding dependency types).  For my vdb,
> > I have 15523 across 798 pkgs.  1331 of that is *DEPEND, converted to 
> > DEPENDENCIES the file count is 748.  Note that's preserving DEPEND, 
> > although it's worthless at this stage of the vdb.  So 5% reduction in 
> > files in there.  Whoopy-de-doo, right?
> 
> So we can modify vdb now? What about all those applications which
> obviously are broken due to that?

You're misunderstanding the problem of the VDB.  We cannot change the 
core structure of it- certain ebuilds currently expect to be able to 
find USE/IUSE at specific pathways.  That's where we're blocked.

Adding a new metadata key can, and has been done (defined_phases, 
properties, required_use, etc).  Removing keys can be done.  For 
efficiency, not writing a key if it's empty, can be and is being done.

Basically, your retort again, is wildly off mark and misunderstanding 
the problem.

I strongly suggest you go do some real research into the existing 
PMs, and the problems being discussed.  Go read some code; you've 
minimally got 3 different codebases to work from, and that's not 
counting ML archives, tools, PMS, or the devmanul.

If per you're statements, you're actually serious about writing your 
own PM, you're going to need to know this sort of shit /anyways/ to 
actually do the basic legwork of a PM, so please go do the necessary 
legwork if you want to persist in arguing about internals.

~harring

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

On Sun, 16 Sep 2012 04:49:21 -0700
Brian Harring <ferringb@gmail.com> wrote:

> On Sun, Sep 16, 2012 at 01:21:26PM +0200, Micha?? G??rny wrote:
> > On Sun, 16 Sep 2012 04:10:01 -0700
> > Brian Harring <ferringb@gmail.com> wrote:
> > 
> > > On Sun, Sep 16, 2012 at 09:56:27AM +0200, Micha?? G??rny wrote:
> > > > But consider that for example Zac & AxS (correct me if I recall
> > > > it correctly) considered making changing the meaning of RDEPEND
> > > > to install them before the build, thus effectively making
> > > > 'build,run' useless.
> > > 
> > > I really am not trying to be a blatant dick to you, but this has 
> > > /zero/ relevance.  RDEPEND means "required for runtime".  That
> > > ain't changing.  If they were discussing changing what RDEPEND
> > > meant, then they were high, period.
> > > 
> > > If zac/axs want to try and make the resolver install RDEPEND
> > > before DEPEND... well, they're free to.  That doesn't change the
> > > fact that the deps still must be specified correctly; in short,
> > > build,run is very much relevant.
> > 
> > I don't think we have made up our mind what *exactly* we want from
> > deps.
> 
> Are we now expecting deps to give us ponies or something?  We know 
> *exactly* what we want from deps, and their current definition- the 
> problem isn't the definition, it's that we don't have the forms we 
> need.

No, the problem is that we think we need more than we have now. Unless
you're considering the whole point of this thread is cosmetics... then
please leave that to Fedora or other people who are paid to change
stuff just because they can.

> > Just because we have something semi-correct right now, doesn't
> > mean that we don't want to change that.
> 
> This is a no-op argument against the proposal: "we can't 
> change the deps because we might want to change the deps".  It's also 
> irrelevant due to the core basis of it being broken as fuck
> (described above).

What I'm trying to say is that you're making a lot of noise about
cosmetics while we haven't even agreed on what's supposed to be inside.
So, are we introducing this obtuse syntax for three DEPEND variables,
of which the third is almost never used?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Brian Harring]

On Sun, Sep 16, 2012 at 03:39:22PM +0800, Ben de Groot wrote:
> On 16 September 2012 09:20, Brian Harring <ferringb@gmail.com> wrote:
> > Dumps are at
> > http://dev.gentoo.org/~ferringb/unified-dependencies-example/ .
> >
> > Herds, if you want to see what your pkgs would look like, look at
> > http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ .
> >
> > If you'd like to see an *example effect* it has on what gets displayed
> > to the user (aka, after all major use conditionals are stripped), look
> > at
> > http://dev.gentoo.org/~ferringb/unified-dependencies-example/user-visible.txt
> > ; warning, that's a 55MB file.  The syntax in use there isn't great,
> > but as said, it's an example.
> >
> > ...
> >
> > Additionally, the form used here makes *no assumption about default
> > context*; in any final solution we use, a default context would be
> > wise- say build,run.  Again, an example of what I mean.

The dumps were regenerated; a default context of 'build,run' was 
added.  Basically in the absense of an explicit dep: targetting, 
dep:build,run is assumed.

Essentially it makes the deps cleaner to read for the common case, 
while also reducing the footprint at the cache, and vdb level; see
http://dev.gentoo.org/~ferringb/unified-dependencies-example/vdb-effect.txt 


> Thanks. I have given it a quick overview for the qt herd. I still
> don't see what using DEPENDENCIES adds to what we do now with separate
> *DEPEND variables. I see no convincing reason to change what we do.

Ok, so here's some stats: in the tree, we have 31360 ebuilds, and 194 
eclasses; grand total of 31554 sources of metadata content (I say 
metadata since vapier has eblits in use which are just phase 
functions- I'm not scanning those intentionally).

Doing some simple scans of the tree, here's some stats; note these 
stats are duplicated in the glep (they're nice selling points, thus 
might as well):

1) 746 hits in the tree for COMMON_DEPEND; that's 2%, and the usages 
I'm aware of have been for literally, what it sounds like- depends 
that are both DEPEND and RDEPEND.

2) scanning for assignments of RDEPEND=.*|$\{?DEPEND\}? gets a hit on 
5343 unique sources.  Searching for the inverse gets a hit on 10008 
unique sources.  Meaning that right there, ~48.6% of the tree is 
duplicating deps between the two forms.  This puts us to 16083 unique 
sources in the tree that would benefit in some form (~51%).

3) What's interesting about that 51% is the eapi groupings; in EAPI4,
the autosetting of RDEPEND="${RDEPEND:-${DEPEND}}" was discontinued.  
Roughly 50% of the initial 51% match is EAPI4; the rest are eapi0-3.

4) Again, keep in mind that the grep's in use above are single line 
matches- I'm definitely missing some ebuilds, and for complex 
dependencies that are appended/set by the eclass, likely missing a lot 
of that too.

So... basically, people are already doing this manually with their own 
intermediate vars.

Just a rough but mildly entertaining stat there's basically 8.38MB 
worth of normalized, literal fricking dependency; using a crappy algo 
(rather than a human doing the deps who can do it better), 37.4% 
(3.1MB) of that is removed via going to dependencies.  It goes 
without saying that this would be a helluva lot less torturous on a 
proper PM implementation that parses the tree once, and renders- let 
alone repoman being able to avoid repeat scans of things it already 
has examined.

Mind you, portage doesn't do that, but this would be good incentive to 
do a proper tree. :)


> As I've said before on IRC, we need a good costs/benefits overview.
> Right now I only see costs (migrating ebuilds and eclasses) and no
> benefits.

Offhand... I wouldn't be pushing for this if I didn't think this would 
be a boon over all- both for devs, and PMs.

I think the actual 'cost' probably got lost in the noise of the 
various flames; what I'm proposing is basically zero cost for devs, it 
shoves the work to the PM, leaving the option for devs to do a more 
fine grained form if they can.

I'm starting a seperate thread w/ a glep for this; I think you should 
take a look at the exact details of the specification including how
DEPEND/RDEPEND/PDEPEND are handled in parallel to DEPENDENCIES (short 
version: if existent, they're automatically folded into DEPENDENCIES 
in my proposal); the end result is basically zero pain transition, 
while enabling us to start getting gains.

Cheers-
~brian

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Brian Harring]

On Sun, Sep 16, 2012 at 02:02:24PM +0200, Micha?? G??rny wrote:
> On Sun, 16 Sep 2012 04:49:21 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> 
> > On Sun, Sep 16, 2012 at 01:21:26PM +0200, Micha?? G??rny wrote:
> > > On Sun, 16 Sep 2012 04:10:01 -0700
> > > Brian Harring <ferringb@gmail.com> wrote:
> > > 
> > > > On Sun, Sep 16, 2012 at 09:56:27AM +0200, Micha?? G??rny wrote:
> > > > > But consider that for example Zac & AxS (correct me if I recall
> > > > > it correctly) considered making changing the meaning of RDEPEND
> > > > > to install them before the build, thus effectively making
> > > > > 'build,run' useless.
> > > > 
> > > > I really am not trying to be a blatant dick to you, but this has 
> > > > /zero/ relevance.  RDEPEND means "required for runtime".  That
> > > > ain't changing.  If they were discussing changing what RDEPEND
> > > > meant, then they were high, period.
> > > > 
> > > > If zac/axs want to try and make the resolver install RDEPEND
> > > > before DEPEND... well, they're free to.  That doesn't change the
> > > > fact that the deps still must be specified correctly; in short,
> > > > build,run is very much relevant.
> > > 
> > > I don't think we have made up our mind what *exactly* we want from
> > > deps.
> > 
> > Are we now expecting deps to give us ponies or something?  We know 
> > *exactly* what we want from deps, and their current definition- the 
> > problem isn't the definition, it's that we don't have the forms we 
> > need.
> 
> No, the problem is that we think we need more than we have now.

Read what I wrote.  "we don't have the forms we need"; a more proper 
statement is "we don't have all of the forms we need".

Please read what I write, rather than just responding blindly.  You 
may have time to waste, but I don't, nor do the people on this ml need 
to see you respond 13 minutes after I send an email when you 
can't even be bothered to read the fucking content properly.


> Unless
> you're considering the whole point of this thread is cosmetics... then
> please leave that to Fedora or other people who are paid to change
> stuff just because they can.

This isn't productive; frankly it's childish.  Take it to the forums 
if you want to continue on tangents like this.


> > > Just because we have something semi-correct right now, doesn't
> > > mean that we don't want to change that.
> > 
> > This is a no-op argument against the proposal: "we can't 
> > change the deps because we might want to change the deps".  It's also 
> > irrelevant due to the core basis of it being broken as fuck
> > (described above).
> 
> What I'm trying to say is that you're making a lot of noise about
> cosmetics while we haven't even agreed on what's supposed to be inside.
> So, are we introducing this obtuse syntax for three DEPEND variables,
> of which the third is almost never used?

Reiterating the points again, and for the final time for you since you 
seem intent on riding the short bus for this particular subject:

1) This unifies the existing syntax down into a collapsed form.  In 
doing so, there are measurable gains across the board for PM 
efficiency and rsync alone.

2) In unifying the syntax via reusing our /existing fucking syntax/, 
we formalize the adhoc common dependency assignments devs already are 
doing in the tree.

3) In moving to a unified syntax, it positions us to easily introduce 
new dependency types without introducing more redundancy.  Easier to 
add new dep types, faster to add new dep types, more efficient in 
doing so in comparison to existing approaches, and done in a fashion 
that devs can reuse existing conditionals.

4) It is not exherbo's DEPENDENCIES.  Meaning it is not label based.  
Meaning you do not need to knee-jerk attack it because of some notion 
it's ciaran based/related.

Honestly, stop wasting my (and others time) and please read this email 
full and through, including the /full thread you're blindly 
responding to/ before responding again.

There is no prive for having the fastest turn around time in 
responding to an email; not unless you consider a permenant /ignore 
and killfile addition to be a prize.

~harring

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Matt Turner]

On Sun, Sep 16, 2012 at 6:15 AM, Brian Harring <ferringb@gmail.com> wrote:
> 1) 746 hits in the tree for COMMON_DEPEND; that's 2%, and the usages
> I'm aware of have been for literally, what it sounds like- depends
> that are both DEPEND and RDEPEND.

CDEPEND is pretty common as well. I could 466 files with CDEPEND.

Re: [gentoo-dev] example conversion of gentoo-x86 current deps to unified dependencies

[Ben de Groot]

On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com> wrote:
> On Sun, Sep 16, 2012 at 03:39:22PM +0800, Ben de Groot wrote:
>> Thanks. I have given it a quick overview for the qt herd. I still
>> don't see what using DEPENDENCIES adds to what we do now with separate
>> *DEPEND variables. I see no convincing reason to change what we do.
>
> Ok, so here's some stats: in the tree, we have 31360 ebuilds, and 194
> eclasses; grand total of 31554 sources of metadata content (I say
> metadata since vapier has eblits in use which are just phase
> functions- I'm not scanning those intentionally).
>
> Doing some simple scans of the tree, here's some stats; note these
> stats are duplicated in the glep (they're nice selling points, thus
> might as well):
>
> 1) 746 hits in the tree for COMMON_DEPEND; that's 2%, and the usages
> I'm aware of have been for literally, what it sounds like- depends
> that are both DEPEND and RDEPEND.
>
> 2) scanning for assignments of RDEPEND=.*|$\{?DEPEND\}? gets a hit on
> 5343 unique sources.  Searching for the inverse gets a hit on 10008
> unique sources.  Meaning that right there, ~48.6% of the tree is
> duplicating deps between the two forms.  This puts us to 16083 unique
> sources in the tree that would benefit in some form (~51%).
>
> 3) What's interesting about that 51% is the eapi groupings; in EAPI4,
> the autosetting of RDEPEND="${RDEPEND:-${DEPEND}}" was discontinued.
> Roughly 50% of the initial 51% match is EAPI4; the rest are eapi0-3.
>
> 4) Again, keep in mind that the grep's in use above are single line
> matches- I'm definitely missing some ebuilds, and for complex
> dependencies that are appended/set by the eclass, likely missing a lot
> of that too.
>
> So... basically, people are already doing this manually with their own
> intermediate vars.

And this works fine, so it doesn't warrant a cosmetic change.

> Just a rough but mildly entertaining stat there's basically 8.38MB
> worth of normalized, literal fricking dependency; using a crappy algo
> (rather than a human doing the deps who can do it better), 37.4%
> (3.1MB) of that is removed via going to dependencies.  It goes
> without saying that this would be a helluva lot less torturous on a
> proper PM implementation that parses the tree once, and renders- let
> alone repoman being able to avoid repeat scans of things it already
> has examined.
>
> Mind you, portage doesn't do that, but this would be good incentive to
> do a proper tree. :)

PM internals are not really relevant to us developers. Just make it work... ;-)

The current *DEPEND system works. So again, I don't see a convincing
reason for change.

>> As I've said before on IRC, we need a good costs/benefits overview.
>> Right now I only see costs (migrating ebuilds and eclasses) and no
>> benefits.
>
> Offhand... I wouldn't be pushing for this if I didn't think this would
> be a boon over all- both for devs, and PMs.
>
> I think the actual 'cost' probably got lost in the noise of the
> various flames; what I'm proposing is basically zero cost for devs, it
> shoves the work to the PM, leaving the option for devs to do a more
> fine grained form if they can.

As soon as you expose this to devs (ebuild/eclass developers and
maintainers), you can no longer talk about zero cost.

We either have the enormous cost of migrating the whole tree to the
new system — which I understand you are not advocating because of the
cost; or we have the (lower, but not zero) cost of maintaining two
systems in the tree, which developers will have to know and understand
in order to be able to deal with them correctly. The latter will also
make eclasses needlessly more complex.

So in my opinion it is folly either way, because the costs are high,
and the benefits only cosmetic.

> I'm starting a seperate thread w/ a glep for this; I think you should
> take a look at the exact details of the specification including how
> DEPEND/RDEPEND/PDEPEND are handled in parallel to DEPENDENCIES (short
> version: if existent, they're automatically folded into DEPENDENCIES
> in my proposal); the end result is basically zero pain transition,
> while enabling us to start getting gains.

Which gains? The glep only talks about cosmetics...

-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

[gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies

[Duncan]

Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as excerpted:

> On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com> wrote:

>> So... basically, people are already doing this manually with their own
>> intermediate vars.
> 
> And this works fine, so it doesn't warrant a cosmetic change.

@ferringb:

yngwin has a point that I've not seen addressed.

What /is/ wrong with the whole CDEPEND intermediate var idea?  It seems 
to work and /I/ don't know of any problems with it (and it would appear, 
neither does yngwin), yet you talk about it as if there's something wrong 
with it.

And while we're at it, do DEPEND="$RDEPEND ..." style solutions have the 
same problems (or lack thereof)?

FWIW I personally like the whole single-var idea, and CERTAINLY 
appreciate the various statistical cache savings, etc.  If we were 
starting from scratch now, I'd definitely favor the single var approach.  
But the combined developer mental cost of having to learn the new method 
and then maintain a working understanding of both over some longer period 
is nothing to sneeze at, and I'm not entirely convinced that it's worth 
that cost, even assuming a doubling of the number of dependency types 
with a lot of commonality between them, and the added benefit a single 
deps var would have in that case.

And the case for a single deps var isn't being helped by the implication 
that there's something wrong with both the intermediate var and copying 
var methods, without ever saying what that "wrong" might be, in the face 
of the experience of many that those existing methods "just work".  So if 
there's something wrong with them, let's get it out there where people 
can see it.  And if there isn't, please eliminate the noise of that 
implication from the argument.

Thanks. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies

[Michael Orlitzky]

On 09/19/2012 06:59 AM, Duncan wrote:
> Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as excerpted:
> 
>> On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com> wrote:
> 
>>> So... basically, people are already doing this manually with their own
>>> intermediate vars.
>>
>> And this works fine, so it doesn't warrant a cosmetic change.
> 
> @ferringb:
> 
> yngwin has a point that I've not seen addressed.
> 
> What /is/ wrong with the whole CDEPEND intermediate var idea?  It seems 
> to work and /I/ don't know of any problems with it (and it would appear, 
> neither does yngwin), yet you talk about it as if there's something wrong 
> with it.
> 
> And while we're at it, do DEPEND="$RDEPEND ..." style solutions have the 
> same problems (or lack thereof)?

The problem appears as we introduce more DEPEND variables (which is what
prompted the proposal, IIRC). If we have ADEPEND, BDEPEND, CDEPEND, and
DDEPEND, and there's only some (i.e. not total) sharing going on then
the COMMON_DEPEND pattern starts to fall apart. You potentially need,

  AB_DEPEND
  AC_DEPEND
  AD_DEPEND
  BC_DEPEND
  BD_DEPEND
  CD_DEPEND
  ABC_DEPEND
  ABD_DEPEND
  ACD_DEPEND
  BCD_DEPEND
  ABCD_DEPEND (COMMON_DEPEND)

This obviously gets worse as more DEPEND vars are introduced.

Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 19/09/12 09:09 AM, Michael Orlitzky wrote:
> On 09/19/2012 06:59 AM, Duncan wrote:
>> Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as
>> excerpted:
>> 
>>> On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com>
>>> wrote:
>> 
>>>> So... basically, people are already doing this manually with
>>>> their own intermediate vars.
>>> 
>>> And this works fine, so it doesn't warrant a cosmetic change.
>> 
>> @ferringb:
>> 
>> yngwin has a point that I've not seen addressed.
>> 
>> What /is/ wrong with the whole CDEPEND intermediate var idea?  It
>> seems to work and /I/ don't know of any problems with it (and it
>> would appear, neither does yngwin), yet you talk about it as if
>> there's something wrong with it.
>> 
>> And while we're at it, do DEPEND="$RDEPEND ..." style solutions
>> have the same problems (or lack thereof)?
> 
> The problem appears as we introduce more DEPEND variables (which is
> what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
> CDEPEND, and DDEPEND, and there's only some (i.e. not total)
> sharing going on then the COMMON_DEPEND pattern starts to fall
> apart. You potentially need,
> 
> AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND 
> ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
> (COMMON_DEPEND)
> 
> This obviously gets worse as more DEPEND vars are introduced.
> 

Well not really, no -- the additional *DEPENDs that are being proposed
(or at least mentioned) for new EAPI will either remove atoms from
COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
COMMON_DEPEND or other intermediate variable won't really be necessary
for them.

Besides, this isn't actually a -problem- as there's nothing which
really requires one to use such helpers; ebuild writers just, well,
can.  :)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBZxZIACgkQ2ugaI38ACPDp4wD/atjvaOsi/ntDMB1Dj7lSAVmW
45qKz6+OO+H/+6eFeVIA/Rz0s7FiG6d2frboHXpYrDBzM1FZcU85AqZti34tR8+h
=E78Z
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies

[Brian Harring]

Pardon the belated response; responding to emails that are quick where 
possible, but lagging on -dev.  Missed this one however...

On Wed, Sep 19, 2012 at 09:16:02AM -0400, Ian Stakenvicius wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 19/09/12 09:09 AM, Michael Orlitzky wrote:
> > On 09/19/2012 06:59 AM, Duncan wrote:
> >> Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as
> >> excerpted:
> >> 
> >>> On 16 September 2012 21:15, Brian Harring <ferringb@gmail.com>
> >>> wrote:
> >> 
> >>>> So... basically, people are already doing this manually with
> >>>> their own intermediate vars.
> >>> 
> >>> And this works fine, so it doesn't warrant a cosmetic change.
> >> 
> >> @ferringb:
> >> 
> >> yngwin has a point that I've not seen addressed.
> >> 
> >> What /is/ wrong with the whole CDEPEND intermediate var idea?  It
> >> seems to work and /I/ don't know of any problems with it (and it
> >> would appear, neither does yngwin), yet you talk about it as if
> >> there's something wrong with it.
> >> 
> >> And while we're at it, do DEPEND="$RDEPEND ..." style solutions
> >> have the same problems (or lack thereof)?
> > 
> > The problem appears as we introduce more DEPEND variables (which is
> > what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
> > CDEPEND, and DDEPEND, and there's only some (i.e. not total)
> > sharing going on then the COMMON_DEPEND pattern starts to fall
> > apart. You potentially need,
> > 
> > AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND 
> > ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
> > (COMMON_DEPEND)
> > 
> > This obviously gets worse as more DEPEND vars are introduced.
> > 
> 
> Well not really, no -- the additional *DEPENDs that are being proposed
> (or at least mentioned) for new EAPI will either remove atoms from
> COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
> COMMON_DEPEND or other intermediate variable won't really be necessary
> for them.

It depends on the dep type actually, and how we're viewing those deps- 
if they must be complete or not.

Consider test depends for example.  Either we can specify it as "in 
addition to depend"- which sucks, because it's entirely possible for a 
DEPEND target to not be required for TDEPEND.

While that's a corner case, it's actually easy to address; just 
require TDEPEND to be comprehensive for the test phase.

Thus DEPEND and TDEPEND suddenly share a lot, and TDEPEND shares part 
of RDEPEND.

Moving on to a real world example... PDEPEND.  Currently a full graph 
resolution requires pulling RDEPEND and PDEPEND, collapsing them, and 
ensuring they're fullfilled for anything that is already installed 
(ie, PDEPEND is required after the transaction is completed).

In our current separated var setup, this as said, this requires the PM 
to track/handle it separately.  However, were PDEPEND to be 
complete/full- that would mean the PM could just render for dep:post 
and know "this is what is necessary outside of the transactional block 
of building/installing it".

So RDEPEND and PDEPEND actually share a *shitton*- they're the same 
contents in 97% of the tree.  Meaning dep:run,post? for 97% of the 
tree, w/ 3% needing to have an addition dep:post? section.

Offhand, comprehensive deps make things easier for devs- it gives them 
the ability to be crystal-freaking-clear as to what's needed at each 
stage; via them doing so, it means the resolver has a greater space to 
dig itself out of fucked up situations if necessary.

Additionally, it actually makes life easier for PM authors.  If we had 
TDEPEND (test), this is how things would go; note this is written for 
just an install, not a replacement (replace is similar, just noisier 
deflecting from my point):

1) ensure DEPEND is satisfied
2) run phases setup -> compile
3) ensure TDEPEND is satisfied;  Implicitly, DEPEND in the process (if 
there is a cycle induced by TDEPEND + DEPEND being required, we have 
no way out of it).
4) run install phase
5) ensure RDEPEND is satisfied.  No longer care about TDEPEND/DEPEND.
6) run pkg_preinst, merge, pkg_postinst.
7) a transaction is opened up, that must be completed before the PM 
exits; before that exit, RDEPEND must still be satisfied, as must 
PDEPEND.

With what I'm suggesting.
1) ensure dep:build is satisfied
2) run phases setup -> compile
3) Ensure dep:test is satisfied.  If cycle breaking is necessary, 
anything in dep:build but not dep:test can be pulled.
4) install phase
5) ensure dep:run is satisfied.  Again, if cycle breaking is 
necessary, punt whatever isn't in dep:run as necessary to break that 
cycle.
6) pkg_preinst, merge, pkg_postinst.
7) transactional block; ensure dep:post is satisfied before exiting 
said transaction.

That's for install; for the PM considering a pre-installed pkg, it 
converts from;
1) ensure RDEPEND and PDEPEND are satisfied

to
1) ensure dep:post is satisfied.

Tbh, I'm not sure I'm communicating these gains as well as I could be; 
at first glance, I'm pretty sure people will say "what's the 
difference?" because the gains/difference there are subtle.

From a PM level, I can tell you this shift to complete deps per stage 
*does* make things easier via simplifying the rules at each step 
(namely, render for this form, that's what's required for this point 
on), while giving full control to devs to be dead on accurate in their 
deps specified (which the PM can then exploit for better cycle 
breaking).

The point I'm trying to make here is that each dep phase 
should be authorative; in doing so, you start getting 
a lot of potential subsets (DEPEND is a subset of TDEPEND, TDEPEND 
isn't completely, but mostly a subset of RDEPEND as RDEPEND is a 
likely a superset of DEPEND; PDEPEND is a superset of RDEPEND).

So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in 
the ebuild.  Or you could just use a syntax form that allows you to 
directly inline that up front, rather than having to muck around w/ 
intermediate vars.


> Besides, this isn't actually a -problem- as there's nothing which
> really requires one to use such helpers; ebuild writers just, well,
> can.  :)

Getting to this point; yes, people could hack around it manually.  
Pretty sure that hasn't been in doubt.

The question is what do we *gain* from making them do so.  No one here 
has stated what we gain from COMMON_DEPENDS in comparison to 
DEPENDENCIES (where they can just inline it).

And as I'm trying to lay out, where dependencies *should* be going 
towards, this issue will get worse if we're doing it out of band from 
DEPENDENCIES.

~harring

[gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies

[Duncan]

Brian Harring posted on Sun, 30 Sep 2012 15:15:18 -0700 as excerpted:

> The point I'm trying to make here is that each dep phase should be
> authorative; in doing so, you start getting a lot of potential subsets
> (DEPEND is a subset of TDEPEND, TDEPEND isn't completely, but mostly a
> subset of RDEPEND as RDEPEND is a likely a superset of DEPEND; PDEPEND
> is a superset of RDEPEND).
> 
> So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in the
> ebuild.  Or you could just use a syntax form that allows you to directly
> inline that up front, rather than having to muck around w/ intermediate
> vars.

Thanks /very/ much!  You said you weren't sure you were being clear, but 
this is the first time I've /really/ understood what must surely be the 
root, at any reasonable level at all.

Let me see if I've got it right:

Yes, in some ways all we're dealing with here is "optics", but the 
_problem_ is that with the proposed proliferation in detailed depend-
types, what is now a simple CDEPEND and thus conceptually easy to handle, 
breaks into 10/20/50/whatever-large-number different shards, and what's 
conceptually easy to handle /now/ becomes many many times more difficult 
to handle, both conceptually for package maintainers and practically for 
iterative resolution in the PMs, due to the interplay of all the 
resulting *CDEPENDs.

The proposed solution to that explosion in conceptual complexity not only 
changes the "optics", but by making most of those detail-depend-types 
absolute/authoritative, allows both package managers (the programs, 
machine) and package maintainers (the humans) to consider each depend-
type separately, thus decreasing both conceptual complexity to a once 
again manageable level for package maintainers (humans), and practical 
complexity for package managers (machine), increasing efficiency, 
reducing resolution time and probably eventually memory/installed-db/
cache size as well.


Of course now I better understand Ciaran's argument for labels as well, 
since it would extend the absolute/authoritative principle even further, 
into the actual deps specification method in ebuilds/eclasses, thereby 
reducing conceptual context load even further via more explicitly 
absolute deps at the local level.

But like you, in practice I don't see that going anywhere in gentoo, in 
the near/short-intermediate future, primarily due to political realities, 
but practically, also due to the conceptual leap it'd require from devs 
(as Ciaran himself points out in response to your statistical analysis of 
exherbo's repo, former gentoo devs simply don't tend to take advantage of 
this aspect of labels in exherbo either; the conceptual leap is in 
practice simply too much).  Thus, while academically, his label approach 
is arguably better in terms of efficiency of absolutes, in practice, 
there's little or no difference between how it's used, and how your 
filtering approach will be used.  Further, given the conceptual distance 
between labels and gentoo's current approach, with filters falling in 
between and political reality, the pragmatic filters approach at least 
has /some/ chance of passing the dev-debate stage and being approved, 
implemented and actually available for use in something like a reasonable 
timeframe (say EAPI-6, in a year's time, a bit more for actual in-tree 
use, given the historic EAPI-a-year processing).  But exherbo style 
labels support altho academically better, given political reality, in all 
likelihood would take at least 2-3 years to pass and be usable in-tree.  
And even then, its practical use as demonstrated in exherbo wouldn't take 
advantage of the differences for another couple years after that, at 
least.

Given that, having use of the useful pragmatic approach in a year's time 
or so seems best, as opposed to an arguably (as you've pointed out, no 
practical demonstration of it in exherbo yet, at least that you've been 
able to find) more academically ideal approach in three, without any real 
benefit over the pragmatic for five or more.

>> Besides, this isn't actually a -problem- as there's nothing which
>> really requires one to use such helpers; ebuild writers just, well,
>> can.  :)
> 
> Getting to this point; yes, people could hack around it manually. Pretty
> sure that hasn't been in doubt.

That's clearer now.  Yes, people could continue to hack around it via 
CDEPENDS, etc.  But the number of common vars (or alternative 
RDEPEND="$DEPEND ..." assignments) and the resulting conceptual load on 
the human maintainer is set to increase exponentially as the number of 
depend-types increases linearly.  At some point it's just no longer 
practically maintainable, and the whole process breaks down.

What the single dependencies variable aims to do in both the filtering 
and labels forms is prevent that ultimate conceptual overload and 
resulting process breakdown by allowing direct compound assignments, thus 
eliminating the intermediate assignments and their exponential 
proliferation.

Thanks again, Brian.  Much clearer now, indeed, at least for me, and 
presumably for others who previously had the same problem I was having.

=:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 30/09/12 06:15 PM, Brian Harring wrote:
> 
> Pardon the belated response; responding to emails that are quick
> where possible, but lagging on -dev.  Missed this one however...
> 

No worries, there's a lot going on.. :D



>>>> yngwin has a point that I've not seen addressed.
>>>> 
>>>> What /is/ wrong with the whole CDEPEND intermediate var idea?
>>>> 
>>> 
>>> The problem appears as we introduce more DEPEND variables
>>> (which is what prompted the proposal, IIRC). If we have
>>> ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some
>>> (i.e. not total) sharing going on then the COMMON_DEPEND
>>> pattern starts to fall apart. You potentially need,
>>> 
>>> AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND 
>>> ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND 
>>> (COMMON_DEPEND)
>>> 
>>> This obviously gets worse as more DEPEND vars are introduced.
>>> 
>> 
>> Well not really, no -- the additional *DEPENDs that are being
>> proposed (or at least mentioned) for new EAPI will either remove
>> atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so
>> tersely that a COMMON_DEPEND or other intermediate variable won't
>> really be necessary for them.
> 
> It depends on the dep type actually, and how we're viewing those
> deps- if they must be complete or not. [ .. Snip! .. ]
> 
> The point I'm trying to make here is that each dep phase should be
> authorative; in doing so, you start getting a lot of potential
> subsets (DEPEND is a subset of TDEPEND, TDEPEND isn't completely,
> but mostly a subset of RDEPEND as RDEPEND is a likely a superset of
> DEPEND; PDEPEND is a superset of RDEPEND).
> 
> So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in
>  the ebuild.  Or you could just use a syntax form that allows you
> to directly inline that up front, rather than having to muck around
> w/ intermediate vars.
> 

... I think what you've just described here might be where the primary
difference in thinking is for most of us, between moving to
DEPENDENCIES and keeping the current *DEPENDs -- to me, from an ebuild
writer's perspective, the *DEPENDS -shouldn't- be authoritative.  IE,
instead of thinking of PDEPEND as a superset of RDEPEND I consider
they are two separate sets, which should not intersect, and are
unioned together to form the full set of runtime dependencies.  IE:
FULL_RUNTIME_DEPEND="$RDEPEND $PDEPEND" somewhere inside portage, if
portage actually needed it this way.

And I see this as how many of the other proposed new *DEPENDs would
work too, ie, they are a refined subset of the bigger sets and should
not intersect with the 'parent' *DEPEND that was used instead on older
EAPIs.

So if this were to change, it might make sense (as Duncan i think
pointed out in his response to this message), to a debate on whether
or not ebuilds must specify an authoritative list for each dep phase.
 (I haven't read through PMS but I'm going to assume that it doesn't
specify this anywhere yet--and if it does, i'm sure Ciaran or someone
will quote it in response :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlBrKKsACgkQ2ugaI38ACPBA9wD/a+XVf2g9s6dcpW1513aXQpYV
tK94QdLkax0Hs6tKFqcA/0+v6oFON2Bd3hedv9DHg7N42NNvtTKK6sOw18OpL0aO
=frmC
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: example conversion of gentoo-x86 current deps to unified dependencies

[Ben de Groot]

> On 30/09/12 06:15 PM, Brian Harring wrote:
>
>>>>> yngwin has a point that I've not seen addressed.
>>>>>
>>>>> What /is/ wrong with the whole CDEPEND intermediate var idea?
>>>>>
>>>>
>>>> The problem appears as we introduce more DEPEND variables
>>>> (which is what prompted the proposal, IIRC). If we have
>>>> ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some
>>>> (i.e. not total) sharing going on then the COMMON_DEPEND
>>>> pattern starts to fall apart. You potentially need,
>>>>
>>>> AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
>>>> ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
>>>> (COMMON_DEPEND)
>>>>
>>>> This obviously gets worse as more DEPEND vars are introduced.
>>>>
>>>
>>> Well not really, no -- the additional *DEPENDs that are being
>>> proposed (or at least mentioned) for new EAPI will either remove
>>> atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so
>>> tersely that a COMMON_DEPEND or other intermediate variable won't
>>> really be necessary for them.

Another thing I wanted to point out is that those "potential" extra
variables are not needed in practice. We already have 98% of the tree
(if I got the previously mentioned stats right) that does fine with
just one or two ({R,}DEPEND). The majority of that other 2% needs just
one more variable. There may be corner cases where more vars would be
needed, but those will never be more than a few ebuilds.

It's just not worth it to completely change the way we do things (or
use two systems in parallel) just for a few ebuilds that would
significantly benefit.

If we were a new distro and designing our ebuild format from scratch,
then yes, I would say your proposal has merit. But we aren't. We have
hundreds of people and tens of thousands of ebuilds using *DEPEND just
fine. There are no big problems, only corner-cases. We're not talking
about incremental improvements either (such as was the case e.g. with
use deps).

Let's just keep things simple, and refrain from "fixing" what isn't broken.
-- 
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin

[gentoo-dev] Re: Re: example conversion of gentoo-x86 current deps to unified dependencies

[Steven J. Long]

On Sun, Sep 30, 2012 at 03:15:18PM -0700, Brian Harring wrote:
> On Wed, Sep 19, 2012 at 09:16:02AM -0400, Ian Stakenvicius wrote:
> > On 19/09/12 09:09 AM, Michael Orlitzky wrote:
> > > The problem appears as we introduce more DEPEND variables (which is
> > > what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
> > > CDEPEND, and DDEPEND, and there's only some (i.e. not total)
> > > sharing going on then the COMMON_DEPEND pattern starts to fall
> > > apart. You potentially need,
> > > 
> > > AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND 
> > > ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
> > > (COMMON_DEPEND)
> > > 
> > > This obviously gets worse as more DEPEND vars are introduced.
> > > 
> > 
> > Well not really, no -- the additional *DEPENDs that are being proposed
> > (or at least mentioned) for new EAPI will either remove atoms from
> > COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
> > COMMON_DEPEND or other intermediate variable won't really be necessary
> > for them.
> 
> It depends on the dep type actually, and how we're viewing those deps- 
> if they must be complete or not.
> 
> Consider test depends for example.

I'd like to see some concrete discussion of link-depends/lib-depends, since
they represent a much more fundamental type of dependency, and would imo
remove atoms from a lot of other places.

Discussing test dependencies first, seems like putting the cart before the
horse.

> The point I'm trying to make here is that each dep phase 
> should be authorative; in doing so, you start getting 
> a lot of potential subsets (DEPEND is a subset of TDEPEND, TDEPEND 
> isn't completely, but mostly a subset of RDEPEND as RDEPEND is a 
> likely a superset of DEPEND; PDEPEND is a superset of RDEPEND).
> 
And all of them are a superset of LDEPEND (which is why I say it's more
fundamental, and class it as an omission from the original conversion
from ports to portage. The fact that link-dependencies are required to be
installed before build can begin, does not make them BUILD_DEPENDS, of
which DEPEND is the equivalent in Gentoo.)

In terms of what has been mentioned before, the set of dependencies
required at a particular point, LDEPEND are required at all of them.

As for the authoritative set at each point, you've already shown that the
package mangler has to build that currently, so there's no implementation
cost in continuing to do so.

> So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in 
> the ebuild.  Or you could just use a syntax form that allows you to 
> directly inline that up front, rather than having to muck around w/ 
> intermediate vars.
>
Or you could address the gaping hole in the spec, and allow people to simply
specify the most fundamental dependency, without having to hack it in.

Then TDEPEND would just be what was required in addition during TEST phases,
a large amount of those common dependency variables would not be needed, and
similarly for C/HOST_DEPENDS (aka the new DEPEND under the awfully-specified
HDEPEND EAPI) since by definition a library dependency lives in the same
ROOT as the installed package.

> 
> > Besides, this isn't actually a -problem- as there's nothing which
> > really requires one to use such helpers; ebuild writers just, well,
> > can.  :)
> 
> Getting to this point; yes, people could hack around it manually.  
> Pretty sure that hasn't been in doubt.
> 
> The question is what do we *gain* from making them do so.  No one here 
> has stated what we gain from COMMON_DEPENDS in comparison to 
> DEPENDENCIES (where they can just inline it).
>
LDEPEND enables much better tracking of linkage, which is pretty fundamental
to building and maintaining a distribution. It's also something that
programmers specify upfront in configure and makefiles. As shown above,
it eliminates a lot of common dependencies, and at least some HDEPEND (if
that variable name is used accurately.)

Additionally, separating out the variable (which yes, can also be done with
a label, but it does need to be done) means linkage can be tracked
automatically based on the sort of data that every package supplies, every
other distribution provides, and every developer has to think about when
writing an ebuild; instead of on yet another syntax obfus^W complication that
was designed for java and python plugins.

Doing it in a separate variable as opposed to labelling it, means that
variable can be treated differently, should that be desired in a new EAPI.
Such a change in how LDEPEND were specified, would not impact on any of the
rest of the single variable format (which is more complex, and more
interdependent, at least in its scripted form) and would not need to be
verified against all the other types of specification in the rest of that
"integrated" variable. So a separate variable is more flexible.

It's also a *lot* more transparent.

For instance, if we'd had that before, do you agree we'd have been scanning
binaries to check for linkage outside that set and system? (After all, we
scan for a lot of other stuff already; scanelf is a gentoo success story.)

How would usage of revdep-rebuild been affected? What about implementation
of preserved-libs?
 
> And as I'm trying to lay out, where dependencies *should* be going 
> towards, this issue will get worse if we're doing it out of band from 
> DEPENDENCIES.
> 
You've given a convincing argument for the package mangler using a unified
variable internally, and if the PM authors all agree, using that in the
cache format. (ie the extra parsing complexity and maintenance is worth it
for the difference in IO across the cache.)

But you've also shown exactly how all that information can be inferred from
the existing ebuild format (and once inferred would actually be simpler than
the whole variable specified in the ebuild.)  As such, it appears more like
something that should be in the back-end, without requiring such a shift in
how people write ebuilds.

Your additional arguments as to why this gets more complex later down the
line are predicated on ignoring LDEPEND.

If that requires a bit more runtime at cache-generation, we're already paying
that price. Saving a small amount of time there, where most users never see
it, doesn't seem worth the mental effort, and maintenance cost, required
to change the format. "Just go ahead and use it in the cache," seems like
the best solution.

Regards,
steveL.