[gentoo-dev] Virtuals - required?

[gentoo-dev] Virtuals - required?

[Jason Stubbs]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I had another crazy idea that I'd like some feedback on. I'll jump straight 
in. With the current virtuals system, there are two basic deficiencies:

1. Packages that can provide a virtual based on USE flags can't. (#32114)
2. Different versions of a virtual are not supported. (#46968)

The proposal for 1 is to have PROVIDE use a syntax similar to DEPEND for 
specifying any USE flag relationship. There is currently no proposal for 2.

What I'm proposing here is to drop virtuals altogether. Instead, I propose 
using meta-packages similar to kde and gnome. To give a give a cut down 
example:

virtual/x11-1.ebuild:
RDEPEND="x86? ( || ( x11-base/xfree x11-base/xorg-x11 ) )
         sparc? ( || ( x11-base/xorg-x11 x11-base/xfree ) )"

The -1 in x11-1 is an arbitrary version. This example shows the only 
deficiency of doing it this way that I am able to think of. Instead of 
specifying the default virtuals per profile, they would only be able to be 
specified per architecture.

Now to an example that shows a resolution to issue 2 above:

virtual/jdk-1.4.ebuild:
RDEPEND="|| ( >=dev-java/blackdown-jdk-1.4 >=dev-java/kaffe-1.1.4 ... )"

The different versions aspect is represented here by kaffe-1.1.4. Even it is 
installed it is presently not able to satisfy >=virtual/jdk-1.4 as its 
version component does not match.

As for issue number 1, it will require a (already scheduled) portage 
enhancement:

virtual/login-1.ebuild:
RDEPEND="|| sys-apps/shadow[pam] sys-apps/pam-login[-pam]"

Here, shadow with USE="pam" or pam-login with USE="-pam" are supported.

As a side note, a single atom is chosen || () dependencies. If something 
installed satisfied any of the atoms, then the installed package is used. 
Otherwise the first package in the list is used.

Finally, there are two additional benefits. One is a slight speed increase as 
portage would no longer have to concern itself with virtuals. Second, is that 
it would be possible to find all packages that satisify a virtual without 
scanning the entire tree for PROVIDEs.

Do I burn in hell? (Again? ;)

Regards,
Jason Stubbs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iQCVAwUBQNQo2FoikN4/5jfsAQL9sQP/UBzFzWlQJ/VkUx+aWGMX505okHhT3XWf
2AxL1y6uTaFdOD4i5kVtzfB+tQ9IOLsVHygIHZS9rFSTC95bvUAX7/yO91BgzIkL
zdI9gKquLgLRexCpBZ5qMLi5Bfruv3bWuCbnU3gTFIjhUSHT2Xs6oHm7jViwTk1W
X9uje3B4xPQ=
=y7Uj
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[joe]

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi all,
>
> I had another crazy idea that I'd like some feedback on. I'll jump
> straight
> in. With the current virtuals system, there are two basic deficiencies:
>
> 1. Packages that can provide a virtual based on USE flags can't. (#32114)
> 2. Different versions of a virtual are not supported. (#46968)
>
> The proposal for 1 is to have PROVIDE use a syntax similar to DEPEND for
> specifying any USE flag relationship. There is currently no proposal for
> 2.
>
> What I'm proposing here is to drop virtuals altogether. Instead, I propose
> using meta-packages similar to kde and gnome. To give a give a cut down
> example:
>
> virtual/x11-1.ebuild:
> RDEPEND="x86? ( || ( x11-base/xfree x11-base/xorg-x11 ) )
>          sparc? ( || ( x11-base/xorg-x11 x11-base/xfree ) )"
>
> The -1 in x11-1 is an arbitrary version. This example shows the only
> deficiency of doing it this way that I am able to think of. Instead of
> specifying the default virtuals per profile, they would only be able to be
> specified per architecture.
>
> Now to an example that shows a resolution to issue 2 above:
>
> virtual/jdk-1.4.ebuild:
> RDEPEND="|| ( >=dev-java/blackdown-jdk-1.4 >=dev-java/kaffe-1.1.4 ... )"
>
> The different versions aspect is represented here by kaffe-1.1.4. Even it
> is
> installed it is presently not able to satisfy >=virtual/jdk-1.4 as its
> version component does not match.

:/ So, say I want my love-sources ebuild in my overlay to provide for
virtual/winkernel (since some versions are patched with win4lin), Rather
then editing the love-sources ebuild (which will survive an emerge sync),
I have to edit this metaebuid for the virtual (which will not survive an
emerge sync unless )

> As for issue number 1, it will require a (already scheduled) portage
> enhancement:
>
> virtual/login-1.ebuild:
> RDEPEND="|| sys-apps/shadow[pam] sys-apps/pam-login[-pam]"
>
> Here, shadow with USE="pam" or pam-login with USE="-pam" are supported.
>
> As a side note, a single atom is chosen || () dependencies. If something
> installed satisfied any of the atoms, then the installed package is used.
> Otherwise the first package in the list is used.
>
> Finally, there are two additional benefits. One is a slight speed increase
> as
> portage would no longer have to concern itself with virtuals. Second, is
> that
> it would be possible to find all packages that satisify a virtual without
> scanning the entire tree for PROVIDEs.
>
> Do I burn in hell? (Again? )

>From portage-*.51

/var/cache/edb/virtuals has been deprecated and is now calculated"
on demand. Strictly _USER_ modifications to virtuals may go into"
/etc/portage/virtuals and will never be modified by portage.

It would sound like there is already work to change virtuals around, and i
assume that "calculated on demand" is pretty much the opposite of your
idea.



--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

On Sat, 19 Jun 2004 20:51:49 +0900 Jason Stubbs <jstubbs@gentoo.org>
wrote:
| What I'm proposing here is to drop virtuals altogether. Instead, I
| propose using meta-packages similar to kde and gnome. To give a give a
| cut down example:
| 
| virtual/x11-1.ebuild:
| RDEPEND="x86? ( || ( x11-base/xfree x11-base/xorg-x11 ) )
|          sparc? ( || ( x11-base/xorg-x11 x11-base/xfree ) )"

For this to work, you'd have to do at least two things:

1) Hack portage to display virtuals stuff differently. Otherwise we'd
end up with reaaaaallyy really messy qpkg / equery / emerge output.

2) Make RDEPEND work. Example::

emerge fluxbox
(watch xorg-x11 get pulled in)
emerge unmerge xorg-x11
(virtual/x11 is still installed)
emerge kde
(watch stuff explode since there're no x libs)

-- 
Ciaran McCreesh : Gentoo Developer (Sparc, MIPS, Vim, Fluxbox)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Virtuals - required?

[Jason Stubbs]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 20 June 2004 00:00, joe@neoturbine.net wrote:
> :/ So, say I want my love-sources ebuild in my overlay to provide for
> virtual/winkernel (since some versions are patched with win4lin), Rather
> then editing the love-sources ebuild (which will survive an emerge sync),
> I have to edit this metaebuid for the virtual (which will not survive an
> emerge sync unless )

Yes, I had considered this and merely failed to mention it in my first post. 
If you are familiar with overlays then you should realize that you would be 
able to copy the meta-ebuild to your overlay and edit it instead. Having an 
"old" version of such an ebuild should not cause any problem as whatever 
virtual it is that is installed should support all required functionality.

(By the way, perhaps love-sources would not get so much ire from developers if 
its users did not always promote it in such an arrogant fashion?)

> From portage-*.51
>
> /var/cache/edb/virtuals has been deprecated and is now calculated"
> on demand. Strictly _USER_ modifications to virtuals may go into"
> /etc/portage/virtuals and will never be modified by portage.
>
> It would sound like there is already work to change virtuals around, and i
> assume that "calculated on demand" is pretty much the opposite of your
> idea.

Yes, I am aware of this. I am on the portage team. One of my motivations for 
this idea is that virtuals add complication to the code where it may not be 
necessary. If all the required virtuals functionality can be moved into the 
normal package code, it simplifies portage further overall and means that 
more bugs can be fixed, more features can be developed and performance can be 
improved further.

Having said that, critism is welcome as long as its intention is to be 
constructive. Any other issues?

Regards,
Jason Stubbs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iQCVAwUBQNRgFloikN4/5jfsAQLAdwP+N8khZf0tKZxeTjloifEprXuihCaK8Scd
WwSqBje1/kgyP5bUHfuXc83sFwRzLQ4UjBgbhixxA3n/DnyM9KfbzkL+09FdHKkw
6gytseeTXi6S9Vl83vysksfolLdNE99uFbsnpN23jayC7fMSIRi9u1sP9gOVAJ0S
KZSJHoaZin4=
=OgRB
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Jason Stubbs]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 20 June 2004 00:04, Ciaran McCreesh wrote:
> On Sat, 19 Jun 2004 20:51:49 +0900 Jason Stubbs <jstubbs@gentoo.org>
> wrote:
> | What I'm proposing here is to drop virtuals altogether. Instead, I
> | propose using meta-packages similar to kde and gnome. To give a give a
> | cut down example:
> |
> | virtual/x11-1.ebuild:
> | RDEPEND="x86? ( || ( x11-base/xfree x11-base/xorg-x11 ) )
> |          sparc? ( || ( x11-base/xorg-x11 x11-base/xfree ) )"
>
> For this to work, you'd have to do at least two things:
>
> 1) Hack portage to display virtuals stuff differently. Otherwise we'd
> end up with reaaaaallyy really messy qpkg / equery / emerge output.

This is not so much of a problem,

> 2) Make RDEPEND work. Example::
>
> emerge fluxbox
> (watch xorg-x11 get pulled in)
> emerge unmerge xorg-x11
> (virtual/x11 is still installed)
> emerge kde
> (watch stuff explode since there're no x libs)

but this means that proper support is a fair way down the track - if possible 
at all. Even if portage prevented (by default) uninstalling packages that had 
others depending on it, an override would still be necessary or else a move 
such as the one from xfree to xorg-x11 would really be hell. ;)

Well, I'll shelve that idea for the present.

Regards,
Jason Stubbs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iQCVAwUBQNRnJ1oikN4/5jfsAQJ7MwP+OWtztVpnwkgTtQCcENNWKzBbjZ6PCwcj
989mkUjAIG9W6BsAyQKAfWouOv6g5BeDQsvQwfx3AuEHszd7b+RNYNwpsjxza5lP
rOM7bHaFXoDF8oQ+ANxKj6enht+y7ODcWmorWURkOBB9+OIxoF6ZoIP7+3gqjuvW
UPVL3DAtif8=
=gxlX
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Jason Stubbs]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 20 June 2004 01:17, Jason Stubbs wrote:
> On Sunday 20 June 2004 00:04, Ciaran McCreesh wrote:
> > On Sat, 19 Jun 2004 20:51:49 +0900 Jason Stubbs <jstubbs@gentoo.org>
> >
> > wrote:
> > | What I'm proposing here is to drop virtuals altogether. Instead, I
> > | propose using meta-packages similar to kde and gnome. To give a give a
> > | cut down example:
> > |
> > | virtual/x11-1.ebuild:
> > | RDEPEND="x86? ( || ( x11-base/xfree x11-base/xorg-x11 ) )
> > |          sparc? ( || ( x11-base/xorg-x11 x11-base/xfree ) )"
> >
> > For this to work, you'd have to do at least two things:
> >
> > 1) Hack portage to display virtuals stuff differently. Otherwise we'd
> > end up with reaaaaallyy really messy qpkg / equery / emerge output.
>
> This is not so much of a problem,
>
> > 2) Make RDEPEND work. Example::
> >
> > emerge fluxbox
> > (watch xorg-x11 get pulled in)
> > emerge unmerge xorg-x11
> > (virtual/x11 is still installed)
> > emerge kde
> > (watch stuff explode since there're no x libs)
>
> but this means that proper support is a fair way down the track - if
> possible at all. Even if portage prevented (by default) uninstalling
> packages that had others depending on it, an override would still be
> necessary or else a move such as the one from xfree to xorg-x11 would
> really be hell. ;)
>
> Well, I'll shelve that idea for the present.

Nope. It's still floating around...

Brian Harring suggested that such an ebuild could have RESTRICT="metaebuild" 
which would signify that just its *DEPENDs should be processed. This would 
easily allow solutions to both of the above problems.

The only other issue I can think of is blocking against virtuals. For example, 
xfree contains DEPEND="!virtual/x11". This could be handled by portage as a 
special case for RESTRICT="metaebuild" but that would then be one point where 
this whole scheme is not compatible with 2.0.50 (and Nick hates special 
cases ;). The only other option would be for virtual/x11 to have:

DEPEND="|| ( ( x11-base/xfree !x11-base/xorg-x11 ) 
             ( x11-base/xorg-x11 !x11-base/xfree ) )"

However, this could get real messy real quick - but does allow for much more 
expression. Thoughts on this?

Regards,
Jason Stubbs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iQCVAwUBQNw0HFoikN4/5jfsAQLXhwP+Is/sY1DI1Lekpe9MBNJGtIu1CHF2N26Z
aAQH3DiPRSxw5qFvWkoMJOZLpt9eAdtIpg8moIGlJns6f8/WTRDunq9fwO2wvgbX
gtxhxl5pgjR3+wBlyq9dlPaBo9ZSIMUCpTv93wLr0zRCtpS6i3c0Vpyc1g5+4iaq
RhIJg7Wv5YA=
=Rwq9
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Jason Stubbs]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

I'll list the pros and cons of this idea. Please let me know if it's 
worthwhile spending my time putting patch together for 2.0.51.

Pros
- ----

1. All current and future atom and depend syntax is available for specifying a 
   virtual. (eg. defaults specifiable based on USE flags, ability to 
   require a set of packages)
2. Versionable virtuals are transparantly supported.
3. Portage speedups as virtuals tracking and checking becomes unnecessary.
4. Can co-exist with the current virtual scheme causing no problems with
   backward compatibility.
5. If implemented before the release of 2.0.51, the cache format change 
   becomes unnecessary (for the time being).

Cons
- ----

1. Default virtuals can only be specified per architecture rather than
   per profile.
2. The virtuals that a package provides are not specified in the package
   itself.

Give me a yes or a no as soon as possible, as 2.0.51-final is not too far away 
and if this goes ahead it'll require at least another _pre version.

Regards,
Jason Stubbs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iQCVAwUBQN1w7VoikN4/5jfsAQJCiAQAkKi+Z5EJ8va7uFNkpMJwm+shtSVRL5Ue
d2LGus60AlhdvQfP6fePcQhFkpNKC+v2EjbjqcuvKZBU/LIUDCfwu/6+NkY+o6Fy
DXBOVjbfaRXJ+Z6fQ88Un8hF099R5rCag0zmfvRkoUwwgVUjcQuvQqahFmVJLp/y
xhYjtjFJiQ8=
=kFSD
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Thomas de Grenier de Latour]

Imho, this change would be a step in the wrong direction, because
it makes portage "less declarative and more imperative". Maybe
this programing paradigms analogy is a bit weird, so I will try to
explain what I mean here. 

For me, ideally, the set of installed packages (and their
configurations, versions, etc.) should depend exactly on a set of
user specifications, and not on the history of user actions (calls
to emerge). With such an ideal portage, it should be possible to
clone a system by installing a stage, copying the /etc/portage
content, and then doing "emerge world" (or "emerge $(cat
/etc/portage/sets/world)" maybe).

This is not currently true for several reasons (i won't detail
them all, that would be offtopic), and will probably never be, but
several recent changes in portage have been made that go exactly
in that direction. For instance, if one uses package.use and
package.keywords files instead of setting some environnement
variables on the command line, then the clone system he will
get using the above described technic will be closer to the
original. It makes things more deterministic and easy to
reproduce.

With that in mind, the introduction of /etc/portage/virtuals seems
to be yet another step in the right direction. With this file,
user can declare his non-default preferences for virtual packages
instanciation. During the initial installation, he can declare
very soon that he prefers Xorg to Xfree, and then call an "emerge
gnome", and that's Xorg that will be installed as a dependency.
Same for cloning a system, if the virtuals file has been copied,
then user's choices will be respected. 

With your solution at the contrary, everything depends on the
order in which packages get installed. If user prefers Xorg, then
he must do an "emerge (--oneshot) xorg-x11" before emerging gnome.
Also, if he wants to clone his system, he has to instanciate
virtuals (all his non-default choices at least) before emerging
world.

I think that the more portage behavior depends on past calls
to emerge history, the less user is in control of his system. At
the contrary, the more it depends on explicit declarations of user
preferences in config files, the more emerge behavior is
predictable and understandable. Now, you may not agree this
statements, in which case my point against your proposal is no
more valid. As i said, that's just my opinion, nothing more.


As a side note, some more random thoughts about your proposal:

 - it would allow to inject virtuals. That could be a good thing.
(I remember that being able to "emerge inject
virtual/linux-sources" had been a request i've seen in the past
on this list.) I don't know how that would play with the
RESTRICT=metaebuild idea tho (depends how you implement it).

 - it would be very handy to have a user tool to list
candidates for a given virtual. Something like "equery
instanciates virtual-pkg-spec".	It is straightforward to implement
with current system (now that the information is available in
aux_get), but seems harder in the context of your proposal.

 - it may make packages cleaning a bit harder: With the old
edb/virtuals system, it was easy to delete a few obsolete
duplicates in the virtuals file, and then ask depclean to finish
the cleanup a safe way. At the contrary, with your system, assume
"virtual/foo" depends on "|| ( bar/pkgA bar/pkgB )", then install
something that depends on virtual/foo (here, pkgA gets installed),
and then install pkgB. Do an "emerge -p depclean", and see
"bar/pkgA" is not listed despite it is not in world nor really
needed anymore. Maybe that can be fixed in depclean code tho, i
don't know. 
 
 - also, i don't know exactly what would be the behavior of
RESTRICT=metaebuild, but if the point is that the package
don't appear at all in /var/db/pkg, then there is an issue:
it makes impossible to track backward dependencies. 
We would then have packages that are:
 * not in world
 * not depended on by anything according to "qpkg -q"
 * not candidate for cleaning according to "emerge depclean"
That would be a bit disturbing for users who wonder "why is this
package there?", and may also be a problem for some external tools
that rely on this backward deps. 


-- 
TGL.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Jason Stubbs]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 27 June 2004 02:32, Thomas de Grenier de Latour wrote:
> With your solution at the contrary, everything depends on the
> order in which packages get installed. If user prefers Xorg, then
> he must do an "emerge (--oneshot) xorg-x11" before emerging gnome.
> Also, if he wants to clone his system, he has to instanciate
> virtuals (all his non-default choices at least) before emerging
> world.

It would be possible to keep /etc/portage/virtuals. Moreso, it would become 
possible to restrict it to virtuals. Try putting "sys-devel/gcc 
sys-libs/glibc" into it and then emerge -p gcc. It will show glibc.

>  - it would allow to inject virtuals. That could be a good thing.
> (I remember that being able to "emerge inject
> virtual/linux-sources" had been a request i've seen in the past
> on this list.) I don't know how that would play with the
> RESTRICT=metaebuild idea tho (depends how you implement it).

I haven't given this any thought, but I guess it could be done.

>  - it would be very handy to have a user tool to list
> candidates for a given virtual. Something like "equery
> instanciates virtual-pkg-spec".	It is straightforward to implement
> with current system (now that the information is available in
> aux_get), but seems harder in the context of your proposal.

Personally, I think it is easier with my proposal. Presently, it requires 
scanning for PROVIDE in every ebuild in the tree. With my proposal, it would 
only require checking the RDEPEND of the metaebuild.

>  - it may make packages cleaning a bit harder: With the old
> edb/virtuals system, it was easy to delete a few obsolete
> duplicates in the virtuals file, and then ask depclean to finish
> the cleanup a safe way. At the contrary, with your system, assume
> "virtual/foo" depends on "|| ( bar/pkgA bar/pkgB )", then install
> something that depends on virtual/foo (here, pkgA gets installed),
> and then install pkgB. Do an "emerge -p depclean", and see
> "bar/pkgA" is not listed despite it is not in world nor really
> needed anymore. Maybe that can be fixed in depclean code tho, i
> don't know.

That is a difficult problem to get around, but exists regardless of how 
virtuals are implemented. It seems like the first argument I've heard for a 
breadth-first rather than depth-first dependency resolver. This issue also 
exists (in a different form) in 2.0.51 but is easy to overcome by explicitly 
stating the desired virtual in /etc/portage/virtuals. The only way to fix it 
with my scheme would be to fix depclean (which is broken in many other ways 
as well...)

Actually, this problem is worse than you've described. app/fooA depends on "|| 
( bar/pkgA bar/pkgB )" and bar/pkgB is installed as a dep from app/fooB when 
app/fooA is emerged. Afterward, bar/pkgA is installed and app/fooB is 
uninstalled. depclean removes bar/pkgB and app/fooA more than likely breaks.

>  - also, i don't know exactly what would be the behavior of
> RESTRICT=metaebuild, but if the point is that the package
> don't appear at all in /var/db/pkg, then there is an issue:
> it makes impossible to track backward dependencies.
> We would then have packages that are:
>  * not in world
>  * not depended on by anything according to "qpkg -q"
>  * not candidate for cleaning according to "emerge depclean"

The main point of RESTRICT="metaebuild" is that it does not show up in emerge 
- -p and that only its RDEPEND is checked. Whether it be installed into the var 
db or not was undecided and something I hadn't given much thought to. I have 
no issue with copying it to the the var db.

Regards,
Jason Stubbs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iQCVAwUBQN4ZI1oikN4/5jfsAQLQ2gP/aQiK+D7cHU4Oyaa/+ISWUgUcVuZKQdhq
vXfte3DSbWdqsPvC1vH1JVth6OT4w/OZhk2Tx/B11UN+J4hXvy4JGN7BzGRBJw23
uRm6g7eaJTuFoXJTopRQIpE3Y7SAlpi2E+MYpguk2gaVfU8jsBH7rRwkgoI60u+O
BreYL9ay8qk=
=v7xi
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Thomas de Grenier de Latour]

On Sun, 27 Jun 2004 09:47:25 +0900
Jason Stubbs <jstubbs@gentoo.org> wrote:

> It would be possible to keep /etc/portage/virtuals.

Maybe that's possible, but i don't see how. The system you've
proposed is much more powerful than the current one, and that's
a problem: a virtual is no more a simple choice between several
possible packages, thus user preferences can no more be a simple
(set of) package(s) choice. 
  To make use of a simple virtuals file like the one currently
available, you would have to limit expressiveness of the
dependency formulae you allow in virtual ebuilds. You must allow
"|| ( A B C )" but forbid "|| ( A && ( B C ) )".
Imo, you can't really use the usual (R)DEPEND but must define a
new kind of formula where only depend atoms and arch? or useflag?
statements operators are allowed. The list of subformulae would be
interpreted as a global disjonction.

> Personally, I think it is easier with my proposal. Presently, it
> requires scanning for PROVIDE in every ebuild in the tree. With
> my proposal, it would only require checking the RDEPEND of the
> metaebuild.

Here again, the problem is the expressiveness of your model. If
you don't require a simple disjonction of dep atoms but allow an
arbitrary depencies formula, it becomes hard to calculate its set
of models. In general case, the best you can answer is the formula
where "arch?" or "useflag?" statements have been interpreted. It
means you have to expect answers of the form "A or (B and C)" for
instance. That was simply impossible with the original virtuals
system, that's why i think it was easier (answers would have
always been of form "A or B or C"). 
  But if you take the simplified depend formulae i've described
above, then i agree it becomes easy.
  That say, what is still hard is to know what virtuals a given
package instanciates (which is required for instance for a correct
implementation of the buildsyspkg feature).

> >  - it may make packages cleaning a bit harder: [snip]
> That is a difficult problem to get around, but exists regardless
> of how virtuals are implemented.

Exists a bit less with <=.50 implementations: with this versions,
it's easy to edit the edb/virtuals file to delete a few obsolete
duplicates, and then depclean works fine to finish the cleanup a
safe but effective way. 
I've not experimented much with current .51_pre tho, so i won't
comment on this one. 

> The only way to fix it with my scheme would be to fix depclean

Yes. It would need addition of special heuristics for
"metaebuilds" dependencies. But for that to work, again, you need
to restrict expressiveness of the dep formula. For instance,
assuming deps are of the form i've described above, then you can
make a depclean that search the smallest solution to this
constraints set:
 - at least one of the possible packages must be kept;
 - if one is in world, then keep at least this one;
 - if one is depended directly by another package, then keep at
least this one;
 - if none of the above two cases is verified, then keep at least
the first packages from the list of candidates.
(or something like that...)
My point is that if packagers start to write virtuals with complex
dependencies formula (like "|| ( A && ( B C ) )"), then this
optimal calculation of what must be kept and what can be cleaned
becomes much more tricky.

> (which is broken in many other ways as well...)

Agreed...

> Actually, this problem is worse than you've described. app/fooA
> depends on "|| ( bar/pkgA bar/pkgB )" and bar/pkgB is installed
> as a dep from app/fooB when app/fooA is emerged. Afterward,
> bar/pkgA is installed and app/fooB is uninstalled. depclean
> removes bar/pkgB and app/fooA more than likely breaks.

Well, yes, that's also bad, but at least this one is right
and optimal from the specified dependencies point of view. In
the case i've described, that was not even true. 
The issue in your example is more yet another proof that binaries
depencies are more strict than ebuilds dependencies can be. That's
close to the openssl 0.9.6 vs 0.9.7 issues (ie. packages can build
against one or the other, but once built they can run only with
the right one). Or did i misunderstood your example?

> Whether it be installed into the var db or not was undecided
> and something I hadn't given much thought to. I have no issue
> with copying it to the the var db.

Imo, it should appear in db, but as a special case (that would
hence not be backward compatible):
 - appear, because it's a necessary link in the backward
dependencies calculation.
 - as a special case, because its presence should not be enough to
satisfy a dependency by itself. Think of emerging something that
depends on virtual/jre, then decide the default jre (lets say
blackdown-jdk) sucks and uninstall it, and then install something
else that depends on virtual/jre... result is a non-working
package. I know that you can't track deep dependencies every time
a package gets installed in current portage. Technicaly, this is
just one more case where a broken package is used to satisfy a
dependency just because it's there. That's nothing new, but it's
much more error prone than with usual packages because user don't
really know about this virtuals and hence is not really
responsible for the mess he introduces when he uninstall, for
instance, blackdown-jdk. 

So what i suggest is that metaebuilds appear in db, but when they
are used in a dep calculation, the algorithm always goes one step 
further(just like with -u) to check whether at least one concrete
packages is really there. And if it's not, then pick the best
concrete package taking /etc/portage/virtuals and ordering of the
depend formula into account (just like if the virtual wasn't
installed). 

-- 
TGL.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Jason Stubbs]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 27 June 2004 12:03, Thomas de Grenier de Latour wrote:
> On Sun, 27 Jun 2004 09:47:25 +0900
> Jason Stubbs <jstubbs@gentoo.org> wrote:
> > It would be possible to keep /etc/portage/virtuals.
>
> Maybe that's possible, but i don't see how. The system you've
> proposed is much more powerful than the current one, and that's
> a problem: a virtual is no more a simple choice between several
> possible packages, thus user preferences can no more be a simple
> (set of) package(s) choice.

When a metaebuild is encountered, check /etc/portage/virtuals for a match. If 
one exists, prepend the settings to the RDEPEND. Maybe embed is a better word 
than prepend. Here is what I'm thinking:

virtual/x11:RDEPEND="|| ( x11-base/xorg-x11 x11-base/xfree )"
portage/virtuals:virtual/x11 x11-base/xfree x11-base/xorg-x11
expand to "|| ( x11-base/xfree  x11-base/xorg-x11 
                || ( x11-base/xorg-x11 x11-base/xfree ) )"

>   To make use of a simple virtuals file like the one currently
> available, you would have to limit expressiveness of the
> dependency formulae you allow in virtual ebuilds. You must allow
> "|| ( A B C )" but forbid "|| ( A && ( B C ) )".
> Imo, you can't really use the usual (R)DEPEND but must define a
> new kind of formula where only depend atoms and arch? or useflag?
> statements operators are allowed. The list of subformulae would be
> interpreted as a global disjonction.

I don't see why the expressiveness must be limited. If there is actually a 
need, I'll throw this idea away as the whole point is code simplification.

> > Personally, I think it is easier with my proposal. Presently, it
> > requires scanning for PROVIDE in every ebuild in the tree. With
> > my proposal, it would only require checking the RDEPEND of the
> > metaebuild.
>
> Here again, the problem is the expressiveness of your model. If
> you don't require a simple disjonction of dep atoms but allow an
> arbitrary depencies formula, it becomes hard to calculate its set
> of models. In general case, the best you can answer is the formula
> where "arch?" or "useflag?" statements have been interpreted. It
> means you have to expect answers of the form "A or (B and C)" for
> instance. That was simply impossible with the original virtuals
> system, that's why i think it was easier (answers would have
> always been of form "A or B or C").
>   But if you take the simplified depend formulae i've described
> above, then i agree it becomes easy.
>   That say, what is still hard is to know what virtuals a given
> package instanciates (which is required for instance for a correct
> implementation of the buildsyspkg feature).

I don't understand the problem you're describing here.

> > >  - it may make packages cleaning a bit harder: [snip]
> >
> > That is a difficult problem to get around, but exists regardless
> > of how virtuals are implemented.
>
> Exists a bit less with <=.50 implementations: with this versions,
> it's easy to edit the edb/virtuals file to delete a few obsolete
> duplicates, and then depclean works fine to finish the cleanup a
> safe but effective way.
> I've not experimented much with current .51_pre tho, so i won't
> comment on this one.

.51_pre removes /var/cache/edb/virtuals altogether and scans
/var/db/pkg/*/*/PROVIDE instead. Hence, the order of the virtuals is 
arbitrary. /etc/portage/virtuals can still be used to do what your describing 
here though.

> > The only way to fix it with my scheme would be to fix depclean
>
> Yes. It would need addition of special heuristics for
> "metaebuilds" dependencies. But for that to work, again, you need
> to restrict expressiveness of the dep formula. For instance,
> assuming deps are of the form i've described above, then you can
> make a depclean that search the smallest solution to this
> constraints set:
>  - at least one of the possible packages must be kept;
>  - if one is in world, then keep at least this one;
>  - if one is depended directly by another package, then keep at
> least this one;
>  - if none of the above two cases is verified, then keep at least
> the first packages from the list of candidates.
> (or something like that...)
> My point is that if packagers start to write virtuals with complex
> dependencies formula (like "|| ( A && ( B C ) )"), then this
> optimal calculation of what must be kept and what can be cleaned
> becomes much more tricky.

See my comment below. The special heuristics are needed already...

> > (which is broken in many other ways as well...)
>
> Agreed...
>
> > Actually, this problem is worse than you've described. app/fooA
> > depends on "|| ( bar/pkgA bar/pkgB )" and bar/pkgB is installed
> > as a dep from app/fooB when app/fooA is emerged. Afterward,
> > bar/pkgA is installed and app/fooB is uninstalled. depclean
> > removes bar/pkgB and app/fooA more than likely breaks.
>
> Well, yes, that's also bad, but at least this one is right
> and optimal from the specified dependencies point of view. In
> the case i've described, that was not even true.
> The issue in your example is more yet another proof that binaries
> depencies are more strict than ebuilds dependencies can be. That's
> close to the openssl 0.9.6 vs 0.9.7 issues (ie. packages can build
> against one or the other, but once built they can run only with
> the right one). Or did i misunderstood your example?

You misunderstood. Take this example from games-board/hexxagon-0.3.1:
DEPEND="|| (
                readline? ( sys-libs/readline )
                gtk? ( =x11-libs/gtk+-1* )
                sys-libs/readline
        )"
This package can build against either sys-libs/readline or x11-libs/gtk+-1* 
but not both. The USE flags complicate it for purposes of the example, but 
almost all ebuilds that use this syntax are similar.

emerge -C sys-libs/readline
emerge =x11-libs/gtk+-1*
USE="-readline gtk" emerge =games-board/hexxagon-0.3.1
emerge sys-libs/readline
USE="-*" emerge depclean

Assuming no other packages come into play, hexxagon is now broken as gtk+ 
would be unmerged. I think there's absolutely no way to get around this other 
than to record what the package was built against. What I was saying earlier 
is that this is a current problem which must be fixed regardless of any 
change to virtuals.

> > Whether it be installed into the var db or not was undecided
> > and something I hadn't given much thought to. I have no issue
> > with copying it to the the var db.
>
> Imo, it should appear in db, but as a special case (that would
> hence not be backward compatible):

It would be backward compatible as long as the virtuals are kept in the 
profiles for some time as well. Portage checks if a package atom is virtual 
before checking any database.

>  - appear, because it's a necessary link in the backward
> dependencies calculation.
>  - as a special case, because its presence should not be enough to
> satisfy a dependency by itself. Think of emerging something that
> depends on virtual/jre, then decide the default jre (lets say
> blackdown-jdk) sucks and uninstall it, and then install something
> else that depends on virtual/jre... result is a non-working
> package. I know that you can't track deep dependencies every time
> a package gets installed in current portage. Technicaly, this is
> just one more case where a broken package is used to satisfy a
> dependency just because it's there. That's nothing new, but it's
> much more error prone than with usual packages because user don't
> really know about this virtuals and hence is not really
> responsible for the mess he introduces when he uninstall, for
> instance, blackdown-jdk.

This is very easy. Just check for RESTRICT="metaebuild".

> So what i suggest is that metaebuilds appear in db, but when they
> are used in a dep calculation, the algorithm always goes one step
> further(just like with -u) to check whether at least one concrete
> packages is really there. And if it's not, then pick the best
> concrete package taking /etc/portage/virtuals and ordering of the
> depend formula into account (just like if the virtual wasn't
> installed).

One other thing as well... /var/db/pkg is not currently used for dep 
calculation. When it does become used, $PORTDIR's copy of the metaebuild 
should be used instead if it exists.

Regards,
Jason Stubbs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iQCVAwUBQN5eHVoikN4/5jfsAQK6igP+PfUJu4R79Jio1B63UBUVBvogqbzY7pUS
MuKxBlQUcXqMBOOvFkX/AufzpXvHjMYaEs5Kizb/leYVUy9+7coq/eLZLSc2v/Hf
7jLMLCVOmCoiu9KJdMZ2O7gdmQHzlSiODTBcSnwo4308z0wNmDN8SQU1YUsHqxJ3
WbxjW5L4q8Q=
=6fug
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Thomas de Grenier de Latour]

On Sun, 27 Jun 2004 14:41:47 +0900
Jason Stubbs <jstubbs@gentoo.org> wrote:

> virtual/x11:RDEPEND="|| ( x11-base/xorg-x11 x11-base/xfree )"
> portage/virtuals:virtual/x11 x11-base/xfree x11-base/xorg-x11
> expand to "|| ( x11-base/xfree  x11-base/xorg-x11 
>                 || ( x11-base/xorg-x11 x11-base/xfree ) )"

That a good idea, but it works well only because the virtual dep
formula is a global disjonction. But what if for instance someone
writes an alsa virtual that looks like this:
  RDEPEND="|| ( media-sound/alsa-driver              
                >=virtual/linux-sources-2.6 )
           media-libs/alsa-lib"
Then the user who wants his driver to be provided by kernel2.6
should probably write this in is virtuals file:
  virtual/alsa  \
     ( >=virtual/linux-sources-2.6 media-libs/alsa-lib )
so that the RDEPEND is transformed into:
 "|| ( >=virtual/linux-sources-2.6 media-libs/alsa-lib )
     ( || ( media-sound/alsa-driver 
            >=virtual/linux-sources-2.6 )
       media-libs/alsa-lib )"

Imho that's a bit too complicated, that's why i think
expressiveness should be limited to a global disjonction. That
said, it may be done by policy instead of being enforced by
the code.

On the same kind of idea, i was thinking about an helper tool for
/etc/portage/virtuals (what ufed is to the USE var). If you are
sure that virtuals dep is just a disjonction, then, for a given
virtual, the tool can present to user the list of possible
choices, and user only have to sort his prefered packages. But if
you allow arbitrary formulae, then such a tool becomes harder
to write and to understand for the user. 

> > That say, what is still hard is to know what virtuals a
> > given package instanciates (which is required for instance for
> > correct implementation of the buildsyspkg feature).

Just to explain this point a bit more, have a look on bug #34964:
The goal was to enable buildpkg for system ebuilds. An issue was
that in profile, there are some system packages specified as
virtuals. The implementation of this "buildsyspkg" feature that is
currently in portage is broken in such cases (if profile ask for
virtual/glibc, the sys-libs/glibc won't be recognized as a system
package). Now that PROVIDES is an aux_get key, that would be easy
to fix. But at the contrary, with your system, this link from a
concrete package to the virtuals it instanciates disappears, so it
can no more be fixed. That's a rather minor issue tho. 

A maybe more important one if PROVIDES disappears is that you
loose some convenient shortcuts when using eclasses. For instance,
for kernel sources, the eclass has PROVIDES=virtual/kernel (and a
few other like this, some of which are conditional, etc.). With
your system, every single kernel sources packages will have to be
listed in the virtual ebuild.

> Assuming no other packages come into play, hexxagon is now
> broken as gtk+ would be unmerged. I think there's absolutely no
> way to get around this other than to record what the package was
> built against. What I was saying earlier is that this is a
> current problem which must be fixed regardless of any change to
> virtuals.

Ok, now i understand better. and you're right that once the
virtuals preferences of the user are integrated in the dep
formula, as you've proposed, then it is no more special case
for depclean.

> > Imo, it should appear in db, but as a special case
>
> This is very easy. Just check for RESTRICT="metaebuild".

Yes, i agree it seems rather straightforward to add this special
case. 


-- 
TGL.

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Jason Stubbs]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 28 June 2004 10:43, Thomas de Grenier de Latour wrote:
>   RDEPEND="|| ( media-sound/alsa-driver
>                 >=virtual/linux-sources-2.6 )
>            media-libs/alsa-lib"
> Then the user who wants his driver to be provided by kernel2.6
> should probably write this in is virtuals file:
>   virtual/alsa  \
>      ( >=virtual/linux-sources-2.6 media-libs/alsa-lib )
> so that the RDEPEND is transformed into:
>  "|| ( >=virtual/linux-sources-2.6 media-libs/alsa-lib )
>      ( || ( media-sound/alsa-driver
>             >=virtual/linux-sources-2.6 )
>        media-libs/alsa-lib )"
>
> Imho that's a bit too complicated, that's why i think
> expressiveness should be limited to a global disjonction. That
> said, it may be done by policy instead of being enforced by
> the code.

We're also trying to prevent portage relying on policy. Policies require a 
strong QA team and that we ain't got. However, I agree that the above is too 
complicated.

The other option is to just treat user's virtual settings as a replacement.
Cases where there are no brackets or any other notation could be treated as 
being inside "|| ()" for backward compatibility.

Actually, you're example above reminded me of bug #25013. Specifically the 
comments toward the end relating to the kde.eclass. As these meta-packages 
are versionable, this scheme would be perfect to handle that (and probably a 
whole lot of other simple eclasses as well).

> On the same kind of idea, i was thinking about an helper tool for
> /etc/portage/virtuals (what ufed is to the USE var). If you are
> sure that virtuals dep is just a disjonction, then, for a given
> virtual, the tool can present to user the list of possible
> choices, and user only have to sort his prefered packages. But if
> you allow arbitrary formulae, then such a tool becomes harder
> to write and to understand for the user.

I've almost finished an API for portage that will make this sort of thing 
relatively easy. If you check it out, the support isn't there yet. By telling 
me what was needed, you just inspired me on how to support it. :)

> > > That say, what is still hard is to know what virtuals a
> > > given package instanciates (which is required for instance for
> > > correct implementation of the buildsyspkg feature).
>
> Just to explain this point a bit more, have a look on bug #34964:
> The goal was to enable buildpkg for system ebuilds. An issue was
> that in profile, there are some system packages specified as
> virtuals. The implementation of this "buildsyspkg" feature that is
> currently in portage is broken in such cases (if profile ask for
> virtual/glibc, the sys-libs/glibc won't be recognized as a system
> package). Now that PROVIDES is an aux_get key, that would be easy
> to fix.

I was aware of the feature and am using it but haven't really looked into the 
implementation. However, there is another issue if it's as you described. 
Dependencies of system packages that aren't listed in system would also be 
exempted. This is a really bad thing, especially if you've built coreutils 
with USE="acl".

I would think that the best way to implement this feature is to look at the 
dependency graph and build a package for anything that was brought in via an 
atom in system.

> But at the contrary, with your system, this link from a concrete package to 
> the virtuals it instanciates disappears, so it can no more be fixed. That's 
> a rather minor issue tho.  

My current implementation of this feature in the API (the end goal is all 
logic in emerge being out and reimplemented) scans each atom in system and 
checks if the package to be merged matches it. Meta-ebuilds shouldn't make 
that any more difficult.

> A maybe more important one if PROVIDES disappears is that you
> loose some convenient shortcuts when using eclasses. For instance,
> for kernel sources, the eclass has PROVIDES=virtual/kernel

True...

> (and a few other like this, some of which are conditional, etc.).

That behaviour is actually broken. There should be no conditionals whatsoever 
in the global scope of eclasses or ebuilds. This is a condition of the 
caching that portage performs and having conditionals in the global scope 
just means that dependencies are calculated using incorrect data.

> With your system, every single kernel sources packages will have to be 
> listed in the virtual ebuild. 

In a way it simplifies it as well, though. For example, many of the kernels 
that inherit kernel-2 and get that automatic PROVIDE have KEYWORDS that 
severely limit the architecture that it can run on. With a ufed-like virtuals 
editor you'd have to check the KEYWORDS of each one to ensure that the 
choices are valid. This way, invalid ones would be gotten rid of in one shot.

Thanks for this discussion, by the way. Most of these issues hadn't entered my 
head whatsoever. If I do make a patch, it'll be well forethought. :)

Regards,
Jason Stubbs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iQCVAwUBQOAW41oikN4/5jfsAQKtJAP9Htz71ff6aeB8a5YW2rlYSrfESOTy3rg6
R2ElRoGi4k+IxNj2pq6pmHvFxv0lHCNgRIqhuhy/FOCYxmyZMqet/VVFs7UUbVqS
nv16HqS+On41rfoCO5mxsoDnkBxMcQp1Dba5YJVZKXNs7AH1Nmbq9kTri/x2oq8j
0Rj0gI7ByLU=
=Xpkx
-----END PGP SIGNATURE-----

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Virtuals - required?

[Thomas de Grenier de Latour]

On Mon, 28 Jun 2004 22:02:24 +0900
Jason Stubbs <jstubbs@gentoo.org> wrote:

> >  "|| ( >=virtual/linux-sources-2.6 media-libs/alsa-lib )
> >      ( || ( media-sound/alsa-driver
> >             >=virtual/linux-sources-2.6 )
> >        media-libs/alsa-lib )"

Oops, what i actually meant was :
  "|| ( ( >=virtual/linux-sources-2.6 media-libs/alsa-lib )
        ( || ( media-sound/alsa-driver
             >=virtual/linux-sources-2.6 )
          media-libs/alsa-lib ) )"

>From $RDEPEND and $USER_PREFS, one would create:
 NEW_RDEPEND="|| ( $USER_PREFS
                   ( $RDEPEND ) )"

> Actually, you're example above reminded me of bug #25013.
> Specifically the comments toward the end relating to the
> kde.eclass. As these meta-packages are versionable, this scheme
> would be perfect to handle that (and probably a whole lot of
> other simple eclasses as well).

You mean, depending on a metaebuild instead of adding some deps
from an eclass? If that's so, agreed, that makes sense. 

> Dependencies of system packages that aren't listed in system
> would also be exempted.

True.

> My current implementation of this feature in the API (the end
> goal is all logic in emerge being out and reimplemented) scans
> each atom in system and checks if the package to be merged
> matches it. 

Sounds perfect. By curosity, how is the cost of this check?
(and do you cache the result of this system deps tree
calculation, to reuse it for the next ebuild?)

> > A maybe more important one if PROVIDES disappears is that you
> > loose some convenient shortcuts when using eclasses. For
> > instance, for kernel sources, the eclass has
> > PROVIDES=virtual/kernel
> 
> True...

This makes me think about an intermediate solution:
 - keep the current PROVIDE in ebuilds syntax, and "virtuals" file
in profile
 - makes emerge generates a set of virtual metaebuilds each time
it rsyncs (using this PROVIDE information plus the virtuals file
in profile)

In terms of fonctionnalities, this system would be almost
equivalent to what currently exists in portage:
 - no need for a kernel dev to edit virtual/kernel.ebuild each
time he makes a new kernel available
 - PROVIDE in eclasses would still be possible
 - virtuals would be only some lists of alternatives, just like it
is currently
 - an ordering of this alternatives can be specified at profile
level profile

Some elements of comparaison with your proposal:
 - it would be less expressive (no more virtuals depending on
sets of packages for instances), but imho this simplicity is also
a good thing
 - it would be almost equivalent in terms of code. The handling in
deps calulation is just the same, and hence it would bring the
same speedup and simplifications. The only difference is the
metaebuilds generator to write, but i don't see major difficulty
here
 - it would need the cache format change that you were avoiding
 - virtual versioning also is possible in this scheme

Well, i'm not sure how good or bad is the idea, it came while
writing this mail, so i've not thought much about it yet.

> > (and a few other like this, some of which are conditional,
> > etc.).
> 
> That behaviour is actually broken.

In the case of kernel.eclass, i think it's working correctly
because the conditions are actually about some statically defined
variables($KV and $ETYPE). Thus, for a given ebuild, it will
always evaluate the same whatever the user config is. 
  At least, this is true as long as bash is used to retrieve this
values. I think i've read that other simpler but faster parsers
may be used in the future, in which case such bash constructions
would become an issue.


> Thanks for this discussion, by the way.

My pleasure :) I like to brainstorm a bit around portage, i just
regret i don't currently have time to also participate with
code... Btw, if you want to continue this discussion, be patient
cause i will be offline next two or three days. 

Regards,

-- 
TGL.

--
gentoo-dev@gentoo.org mailing list