[gentoo-dev] better support for binary packages

[gentoo-dev] better support for binary packages

[Philipp Riegger]

Good morning,

I want to talk about improved binary package support for Gentoo. About
1-2 months ago there already was a discussion about this on gentoo-soc@
and on bugzilla [1]. If I remember correctly, there were no devs
involved in the discussion, so I thought I'll post my thoughts here.

I know, that Gentoo is a source-based distribution or meta-distribution,
and I don't want to make Gentoo another Fedora or Ubuntu, but I think
there are some things we can learn from them.

The current situation:

Binary packages are (usually) stored
in /usr/portage/packages/$category/$package-$version.tbz2. The package
consists of the "real binary package" and the metadata (combined using
xpak or whatever).

Problems I see with this:

1) If a binary package is built because it needs to be linked against a
new library, because the USE-flags change or because the ebuild changes
without a revision bump, the "old" binary package is overwritten. This
also means that there is no support to store multiple packages with
different USE-flags without, well, using different directories.
2) To find out which USE-flags a package is built with, one needs to
download the package and look at the metadata. Today I discoveres a file
called "Packages" which looks like a metadata cache, but I did not find
more information about it (only tried "man portage").

So, how can we address this?

First we should do something about 2), I think: I want to propost the
following scheme:

Binary packages are stored in
$arch/$description/$category/$package/$package-$version-$ev-$use-$bv.tbz2.

$arch: This is x86, ppc or whatever you put into ACCEPT_KEYWORDS minus
 the '~'. It does not make sense to make a distinction here.
$description: Something like pentium3, core2quad, G4, or whatever.
 Pentium3-uclibc, Pentium3-solaris-prefix are also possible.
$category, $package and $version should be clear.
$ev: The "ebuild version". See below.
$bv: The "binary version". See below.
$use: The USE-flags. See below.

About ebuild version, USE-flags and binary version:

I would like to encode the USE-flags into the filename. This enables us
to have binary packages of the same version built with different
USE-flags in the same repository. Some wanted to have this in the
directory, some say it is ok to have it in the xpak only and some prefer
the "Packages"-like file.

I think, USE-flags can be set per package and therefore should be stored
per package, not per $description or whatever. Having it only in the
xpak allows no distinction between multiple binary packages, same
version, differen USE-flags and the same is true for the Packages file.
This would also be created, downloaded all the time and so on. Therefore
I think the cleanest solution is having USE-flags in the filename.

There are different methods to store it there.

a) A checksum (of the USE-flags, the USE-flag string, the ebuild and the
USE-flag string, whatever).
b) List the enabled USE-flags in the filename, use a) if the string gets
too long.
c) Use a packed binary vector.

I don't like a), because it is not easily reversible. You could always
download the Packages file or the binary package and look into the xpak
metadata, but that's too much effort. b) also has the problems i
mentioned for a). Also, you'd need some system to distinguish ebuilds
with the same version but different USE-flags. You also need that for
c), so b) has no advantages ofer c) in my eyes.

For c) I think of the following: Sort the USE-flags in some defined way
(ASCII code, whatever) and make a vector with a 1 for every enabled
USE-flag and a 0 for every disabled USE-flag. Compress that vector: If
you use HEX code, you need 1 character for every 4 bits, but it should
be possible to find 64 different characters, then you need 1 character
for every 6 bits. PHP has 106 USE.flags, that would make a USE-string
with about 18-27 characters. Packages with lots of USE-expand stuff like
languages would need more, but not too much, I think.

Problems: The string might get long, you get big problems with USE-flag
renames, USE-flag additions or removals. That's where the ebuild version
is needed. Or not. We have 3 possibilities:

a) Change policy: USE-flag changes in an ebuild need a version bump. 
b) Use a checksum of the ebuild.
c) Use the version given by the version control system.

The problem with a) is, that is a change in policy and probably hard to
do. Increasing the revision for a (trivial) change leads to a lot of
unnecessary rebuilds for users. It also means, that USE-flag changes in
eclasses are difficult, the eclass should probably copied over to a new
name with version and only ebuilds with a new version (revision) are
allowed to use it. 

The problem with b) is, that it is not ordered. You don't know, which is
the newest version. If you have an ebuild with a version where there is
no binary package for, it gets difficult/ugly.

c) also has problems: When using cvs, there are versions easily
available. The same is true for svn, but lots of distributed version
control systems like git use checksums as versions. Welcome back to b).
Another thing is, how do we get to the versions? Will they be in the
header forever, since they make signing ebuilds or the manifest much
more complicated (multiple commits necessary)? But, well, since metadata
is generated and provided by "the tree", it should be not too hard to ad
a unique ebuild version there (in the case of checksums, use an integer,
increase whenever the checksum changed or something). It just might make
using overlays a bit more difficult.

The last thing to be discribed is the binary version. Lots of people
talk about dependencies to other binary packages when they talk about
binary packages for Gentoo, but that gets quite difficult (and, in my
opinion, ugly). We mostly need to provide a "consistent set" of
packages, which means, if A depends on B, B changes and therefore breaks
A, we need to provide an updated version of A. And we can do that with
simply increasing the binary version, since the package manager knows
then, that this package needs updating, too.

How to create binary packages?

Create some build server (or build server infrastructure). The most
important thing is a script or something that provides the
functionality. One enters a make.conf, /etc/portage dir, path to the
profile, description and whatever else is needed and the system starty
building. Then you can create a second set of data and start building
and the system puts the binary packages in the same directory and
discovers what needs to be built and what not (because apache needs to
be built only once if its USE-flags are the same for the different
configuration sets).

But there are thousands of packages and millions of USE-flag
combinations!

Seriously, who cares? The goal of this project (as it exists in my head)
is not to provide everything. It is to provide the most used packages.
If you need parrot, compile it yourself. If you need netbeans, compile
it yourself. We have @system, gnome, kde and anothe hand full of
packages, which will change over time. I'm, really lookign forward to
the data collected by the statistics project (GSoC).

The same is true for USE-flags: We might provide gnome, kde, both, a
server profile and whatever we decide to provide, but not everthing.
Again, statistics will help.

Same with CFLAGS. Probably no -O3, no -ffast-math, no -break-my-code or
whatever. Probably x86 with 32 and 64 bit for the beginning, later maybe
more.

So, the really really cool thing is, that if you are some company,
university, institution or freak, with lots of (similar) Gentoo boxes,
you can set up a build server and even share the binary packages, if you
want. Same level of security as non-official overlays, but in the
university of FooBar in Jamaica uses it, there should not be too many
security problems.

Thanks for reading, please discuss, I probably forgot lots of stuff, but
I can tell it later in the discussion.

Philipp




[1] https://bugs.gentoo.org/show_bug.cgi?id=150031

Re: [gentoo-dev] better support for binary packages

[lxnay]

[-- Attachment #1: Type: text/plain, Size: 232 bytes --]

This is what I am doing in Sabayon, creating a new layer over Portage => Entropy. I'm almost done, just need to work out some documentation and apidocs.
http://gitweb.sabayon.org/?p=entropy.git;a=summary

-- 
Fabio Erculiani


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 270 bytes --]

[gentoo-dev] Re: better support for binary packages

[Duncan]

lxnay@sabayonlinux.org posted
fv50wm3klntqyetcw4UYAxe124vaj_firegpg@mail.gmail.com, excerpted below, on 
Mon, 25 May 2009 12:16:06 +0200:

> This is what I am doing in Sabayon, creating a new layer over Portage =>
> Entropy. I'm almost done, just need to work out some documentation and
> apidocs. http://gitweb.sabayon.org/?p=entropy.git;a=summary

Agreed.

Gentoo is in general a from-source metadistribution.  There's limited 
support for binary packages in at least one of the package managers 
(portage), but honestly, that's not what Gentoo's best at, and I don't 
believe that will ever change without making it significantly worse at 
what it IS best at now, the normal from-source Gentoo we know and love.

Better to leave the serious distribution level binary repackaging to the 
Gentoo-based distributions like Sabayon.  Let them do what they do best, 
and let Gentoo continue doing what it does best.  By definition, binary 
packaging isn't broken and doesn't need fixed, as that's not part of what 
defines Gentoo, so don't fix what ain't broken! =:^)

That said, I could envision an eselect like "binary profile" switcher, 
that subject to settings in a config file, changes USE flags, CFLAGS, gcc-
configs an appropriate gcc version, etc, changing PKGDIR appropriately as 
well, so one could easily enough create as many "binary profiles" as 
desired and as the use case dictated.  It's likely various reasonably 
large Gentoo deployments are already doing something like this as it 
could certainly be scripted, but an emergable package to make it easy for 
ordinary joe Gentoo user would be useful, and I believe appreciated by 
many.  

(Here, I'd put it to use when testing new gcc versions, making it easy to 
swap out PKGDIRs and revert to the old version either per-package or 
system-wide, if the new version was breaking too much.)

So here:  No to the whole big complicated let's fix Gentoo binaries 
thing.  There's already Gentoo-based binary solutions like Sabayon for 
those so interested, and I can't see Gentoo doing better than they're 
doing, at least not without breaking the from-source that Gentoo's good 
at.  But yes to anyone interested in developing a nice new "binary 
profile" switcher to make managing binary package sets easier for those 
Gentoo admins that would find such a thing useful.  Whether they then 
start making those profiles public and whether anyone else chooses to use 
them is entirely up to the individual admins whose systems would be 
affected.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] better support for binary packages

[Ben de Groot]

lxnay@sabayonlinux.org wrote:
> This is what I am doing in Sabayon, creating a new layer over Portage =>
> Entropy. I'm almost done, just need to work out some documentation and
> apidocs.
> http://gitweb.sabayon.org/?p=entropy.git;a=summary
> 
Only problem with entropy is that it doesn't work nice with portage.

-- 
Ben de Groot
Gentoo Linux developer (qt, media, lxde, desktop-misc)
Gentoo Linux Release Engineering PR liaison
______________________________________________________

Re: [gentoo-dev] better support for binary packages

[lxnay]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]



On Mon, May 25, 2009 at 5:47 PM, Ben de Groot <yngwin@gentoo.org> wrote:
> lxnay@sabayonlinux.org wrote:
>> This is what I am doing in Sabayon, creating a new layer over Portage =>
>> Entropy. I'm almost done, just need to work out some documentation and
>> apidocs.
>> http://gitweb.sabayon.org/?p=entropy.git;a=summary
>>
> Only problem with entropy is that it doesn't work nice with portage.

You are wrong. From entropy 0.95+ all works fine, the only problem previous releases had was with Portage world file being filled incorrectly (emaint --fix world does it, but I am sure you know this).
http://gitweb.sabayon.org/?p=entropy.git&a=search&h=HEAD&st=commit&s=portage

Regards,

>
> --
> Ben de Groot
> Gentoo Linux developer (qt, media, lxde, desktop-misc)
> Gentoo Linux Release Engineering PR liaison
> ______________________________________________________
>
>



-- 
Fabio Erculiani


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 270 bytes --]

Re: [gentoo-dev] better support for binary packages

[Ben de Groot]

lxnay@sabayonlinux.org wrote:
> 
> 
> On Mon, May 25, 2009 at 5:47 PM, Ben de Groot <yngwin@gentoo.org> wrote:
>> lxnay@sabayonlinux.org wrote:
>>> This is what I am doing in Sabayon, creating a new layer over Portage =>
>>> Entropy. I'm almost done, just need to work out some documentation and
>>> apidocs.
>>> http://gitweb.sabayon.org/?p=entropy.git;a=summary
>>>
>> Only problem with entropy is that it doesn't work nice with portage.
> 
> You are wrong. From entropy 0.95+ all works fine, the only problem
> previous releases had was with Portage world file being filled
> incorrectly (emaint --fix world does it, but I am sure you know this).
> http://gitweb.sabayon.org/?p=entropy.git&a=search&h=HEAD&st=commit&s=portage

That's funny, as I just installed Sabayon 4.1 KDE version the other day,
and did an equo world update. After that I wanted to install quassel,
but as the version in equo is ancient, I went for emerge. That wanted to
install qt-core and some other qt packages, which obviously were already
installed as deps for KDE. The emerge failed with a file collision of
qt-core (the newly emerged package with the previous binary package).

I then turned to the #sabayon irc channel and was told:

<yngwin> ok, so you can't just mix equo and emerge?
<Azerthoth> yngwin honestly you can, but you better
 have a real firm grip on how both package systems work,
 and be willing to brick your system
<yngwin> well, i know portage, i dont know equo yet
<Azerthoth> equo is preferred barring that, converting
 the system to portage only, but mixing the two can be
 really tricky
<Azerthoth> not saying you will brick your system, just
 saying that the odds increase exponentially

I would love Sabayon to be Gentoo with binary support, that would have
worked well on my laptop. But if mixing the package managers is so
problematic, that's just not worth the trouble for me.

Cheers,
-- 
Ben de Groot
Gentoo Linux developer (qt, media, lxde, desktop-misc)
Gentoo Linux Release Engineering PR liaison
______________________________________________________

Re: [gentoo-dev] better support for binary packages

[lxnay]

[-- Attachment #1: Type: text/plain, Size: 3050 bytes --]



On Mon, May 25, 2009 at 7:20 PM, Ben de Groot <yngwin@gentoo.org> wrote:
> lxnay@sabayonlinux.org wrote:
>>
>>
>> On Mon, May 25, 2009 at 5:47 PM, Ben de Groot <yngwin@gentoo.org> wrote:
>>> lxnay@sabayonlinux.org wrote:
>>>> This is what I am doing in Sabayon, creating a new layer over Portage =>
>>>> Entropy. I'm almost done, just need to work out some documentation and
>>>> apidocs.
>>>> http://gitweb.sabayon.org/?p=entropy.git;a=summary
>>>>
>>> Only problem with entropy is that it doesn't work nice with portage.
>>
>> You are wrong. From entropy 0.95+ all works fine, the only problem
>> previous releases had was with Portage world file being filled
>> incorrectly (emaint --fix world does it, but I am sure you know this).
>> http://gitweb.sabayon.org/?p=entropy.git&a=search&h=HEAD&st=commit&s=portage
>
> That's funny, as I just installed Sabayon 4.1 KDE version the other day,
> and did an equo world update. After that I wanted to install quassel,
> but as the version in equo is ancient, I went for emerge. That wanted to
> install qt-core and some other qt packages, which obviously were already
> installed as deps for KDE. The emerge failed with a file collision of
> qt-core (the newly emerged package with the previous binary package).

This is gentoo-dev and you are OFF TOPIC.

Btw, that's weird, because on 4.1 I have:

mars / # qlist -ICv | grep qt-
x11-libs/qt-3.3.8b
x11-libs/qt-4.5.0
x11-libs/qt-assistant-4.5.0
x11-libs/qt-core-4.5.0
x11-libs/qt-dbus-4.5.0
x11-libs/qt-gui-4.5.0
x11-libs/qt-opengl-4.5.0
x11-libs/qt-qt3support-4.5.0
x11-libs/qt-script-4.5.0
x11-libs/qt-sql-4.5.0
x11-libs/qt-svg-4.5.0
x11-libs/qt-test-4.5.0
x11-libs/qt-webkit-4.5.0
x11-libs/qt-xmlpatterns-4.5.0

and emerging quassel works as expected.

>
> I then turned to the #sabayon irc channel and was told:
>
> <yngwin> ok, so you can't just mix equo and emerge?
> <Azerthoth> yngwin honestly you can, but you better
>  have a real firm grip on how both package systems work,
>  and be willing to brick your system
> <yngwin> well, i know portage, i dont know equo yet
> <Azerthoth> equo is preferred barring that, converting
>  the system to portage only, but mixing the two can be
>  really tricky
> <Azerthoth> not saying you will brick your system, just
>  saying that the odds increase exponentially

This is told to newbies who mess with USE flags. Since binary is a static world.

>
> I would love Sabayon to be Gentoo with binary support, that would have
> worked well on my laptop. But if mixing the package managers is so
> problematic, that's just not worth the trouble for me.

And this made me definitely wonder about this email :)
Next time, please post Sabayon specific stuff on our ML/com. channels.

Regards,

>
> Cheers,
> --
> Ben de Groot
> Gentoo Linux developer (qt, media, lxde, desktop-misc)
> Gentoo Linux Release Engineering PR liaison
> ______________________________________________________
>
>



-- 
Fabio Erculiani


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 270 bytes --]

Re: [gentoo-dev] better support for binary packages

[Ben de Groot]

lxnay@sabayonlinux.org wrote:

>>>> lxnay@sabayonlinux.org wrote:
>>>>> This is what I am doing in Sabayon, creating a new layer over
>>>>> Portage =>
>>>>> Entropy. I'm almost done, just need to work out some documentation and
>>>>> apidocs.
>>>>> http://gitweb.sabayon.org/?p=entropy.git;a=summary

> Next time, please post Sabayon specific stuff on our ML/com. channels.

Sure, so let's keep entropy out of the discussion.

-- 
Ben de Groot
Gentoo Linux developer (qt, media, lxde, desktop-misc)
Gentoo Linux Release Engineering PR liaison
______________________________________________________

Re: [gentoo-dev] Re: better support for binary packages

[Philipp Riegger]

Hi Duncan,

I don't see the connection between the email Fabio wrote and your
answer. Do you want to say, that you agree that he's doing what i
described and that it works the way i described it? I doubt it. If you
really care, could you answer my first email and state there the
problems you see with the approach and why you think this is making
Gentoo worse?

On Mon, 2009-05-25 at 11:43 +0000, Duncan wrote:
> Gentoo is in general a from-source metadistribution.  There's limited 
> support for binary packages in at least one of the package managers 
> (portage), but honestly, that's not what Gentoo's best at, and I don't 
> believe that will ever change without making it significantly worse at 
> what it IS best at now, the normal from-source Gentoo we know and love.

But how would you make it worse? It already has the functionality to add
repositories for binpackages, why not improve it? Why leave it the way
it is?

> Better to leave the serious distribution level binary repackaging to the 
> Gentoo-based distributions like Sabayon.  Let them do what they do best, 
> and let Gentoo continue doing what it does best.  By definition, binary 
> packaging isn't broken and doesn't need fixed, as that's not part of what 
> defines Gentoo, so don't fix what ain't broken! =:^)

Well actually, some time ago Gentoo was by definition running a linux
kernel or a BSD kernel and now it runs in a prefix on Windows and AIX
and Solaris. Some time ago there was some guy called drobbins who was
kind of the leader of Gentoo and now we have a council. If you really
don't want changes, you could stop running emerge --sync.

> That said, I could envision an eselect like "binary profile" switcher, 
> that subject to settings in a config file, changes USE flags, CFLAGS, gcc-
> configs an appropriate gcc version, etc, changing PKGDIR appropriately as 
> well, so one could easily enough create as many "binary profiles" as 
> desired and as the use case dictated.  It's likely various reasonably 
> large Gentoo deployments are already doing something like this as it 
> could certainly be scripted, but an emergable package to make it easy for 
> ordinary joe Gentoo user would be useful, and I believe appreciated by 
> many.
> 
> (Here, I'd put it to use when testing new gcc versions, making it easy to 
> swap out PKGDIRs and revert to the old version either per-package or 
> system-wide, if the new version was breaking too much.)
> 
> So here:  No to the whole big complicated let's fix Gentoo binaries 
> thing.  There's already Gentoo-based binary solutions like Sabayon for 
> those so interested, and I can't see Gentoo doing better than they're 
> doing, at least not without breaking the from-source that Gentoo's good 
> at.  But yes to anyone interested in developing a nice new "binary 
> profile" switcher to make managing binary package sets easier for those 
> Gentoo admins that would find such a thing useful.  Whether they then 
> start making those profiles public and whether anyone else chooses to use 
> them is entirely up to the individual admins whose systems would be 
> affected.

Not sure, what the binary profile switcher really would change here.
What I was talking about were mostly USE-flags and packages, and I guess
your binary profile switcher doesn't change too much, there.

It would be ok to do all this stuff in an extra project, without Gentoo.
But there are some downsides: You are not Gentoo anymore. The
communication channels get longer and more complicated. In my first post
i described some things that would need to be changed. Either in portage
or in the policy how packages are dealt with. Well, the last is a little
bit impossible outside of Gentoo (I don't want to fork the tree) and I
also don't want to fork portage, so the first is complicated, too.

And all this layer thing Fabio was talking about. I did not try it and I
did not read the code, but I think it makes things much more
complicated. See also the discussion about mixing package managers
between Gentoo and Sabayon. I do not want these problems.

Philipp

Re: [gentoo-dev] better support for binary packages

[Kobboi]

On Mon, 2009-05-25 at 11:54 +0200, Philipp Riegger wrote:

> The current situation:
> 
> Binary packages are (usually) stored
> in /usr/portage/packages/$category/$package-$version.tbz2. The package
> consists of the "real binary package" and the metadata (combined using
> xpak or whatever).

No, the metadata is stored in a separate Packages file.

Regards,
Kobboi

[gentoo-dev] Re: better support for binary packages

[Duncan]

Philipp Riegger <lists@anderedomain.de> posted
1243321504.9661.14.camel@hspc30.informatik.uni-stuttgart.de, excerpted
below, on  Tue, 26 May 2009 09:05:03 +0200:

> I don't see the connection between the email Fabio wrote and your
> answer. Do you want to say, that you agree that he's doing what i
> described and that it works the way i described it? I doubt it. If you
> really care, could you answer my first email and state there the
> problems you see with the approach and why you think this is making
> Gentoo worse?

I agree that an independent approach is the way to go.  Gentoo (or 
rather, its PMs, package managers, all three of them, of which portage is 
only one) doesn't do binaries really well, certainly, but I really don't 
see that changing within Gentoo itself, except at the expense of from-
source, which it does quite well indeed.

> But how would you make it worse? It already has the functionality to add
> repositories for binpackages, why not improve it? Why leave it the way
> it is?

Sure, improve it, but what you are talking about isn't a simple 
improvement, but a massive rearchitecting.  That's not easily doable 
without a chance of focus.  It's that change of focus that would 
eventually kill the quality from-source support we have and that 
continues to develop, now.

>> That said, I could envision an eselect like "binary profile" switcher,
>> that subject to settings in a config file, changes USE flags, CFLAGS,
>> gcc- configs an appropriate gcc version, etc, changing PKGDIR
>> appropriately as well, so one could easily enough create as many
>> "binary profiles" as desired and as the use case dictated.

> Not sure, what the binary profile switcher really would change here.
> What I was talking about were mostly USE-flags and packages, and I guess
> your binary profile switcher doesn't change too much, there.

Sure it does, but incrementally.  Basically, your proposal laid out a 
complicated tree based on metadata, a tree the package manager would have 
to understand and support, what I'm proposing is to allow the same thing, 
but not by adding all that complication to the package manager, but 
rather, by using a newly created application to switch among the branches 
of that tree on request.  Portage (or other PM) would use its configured 
PKGDIR and wouldn't understand the tree, but it wouldn't need to, because 
the switcher would manage switching between the branches according to its 
configuration.

The result would be that in a large enough deployment to have build-
servers, the build server(s) could build a particular set of CFLAGS/USE-
flags/gcc-version/arch/whatever, then switch to another branch and build 
that set.  Individual package clients could simply point at the 
appropriate tree and get most of their packages, at least the common 
ones, that way.

Now this wouldn't directly give you the flexibility of the package name 
changes you proposed, but it could do so indirectly, putting that 
information in the directory tree if configured to do so.  Clients may 
need to use the binprofile switcher as well, for individual packages they 
chose to build outside their normal USE profile, but the process could be 
automated once properly configured, using PM hooks as appropriate.

> It would be ok to do all this stuff in an extra project, without Gentoo.

The proposal above wouldn't be without Gentoo.  It'd simply be a package 
level thing rather than a distribution level thing.  But either that or 
the independent distribution based on Gentoo route that lxnay has taken, 
either one works, without defocusing Gentoo on its meta-distrib-wide from-
source vision.

For that matter, Gentoo already has three package managers, and binary 
support (or the lack thereof) has been deliberately left up to the 
package manager at this point -- it's NOT part of PMS.  It'd be equally 
possible to either take one of the current PMs and add your enhanced 
binary package scheme to it, or to start an independent forth package 
manager, and have it focus on binaries rather than from-source.  (It 
could even take the existing portage binpkgs and rename or otherwise 
manage them as necessary, thereby avoiding the from-source side entirely, 
if so desired.)

> Well, the last is a little bit impossible outside of Gentoo
> (I don't want to fork the tree) and I also don't want to fork portage,
> so the first is complicated, too.

You mention portage, but don't mention the other PMs at all.  As 
mentioned, binary packages aren't part of PMS (the Package Management 
Specification, the app-doc/pms package, the standard that allows PMs 
besides portage, currently, paludis and pkgcore) at all.  Gentoo really 
doesn't have an official binary package format at all (see PMS, Appendix 
B, Unspecified Items), only individual package managers like portage do, 
and at present they may conflict with each other.

That's both a good and a bad thing.  As it's not specified, you're free 
to define it as necessary to meet your needs for any tool you devise to 
handle binary packages.  Of course, that means that it's just that tool's 
definition, and at least one package manager, portage, does happen to 
have binary packages and a working format definition for them.

But that leaves you with maximum flexibility in creating whatever you 
want, either as a separate tool (or even independent Gentoo based 
distribution), or as an expanded PM, either forked from one of the 
current ones, or created independently.
 
> And all this layer thing Fabio was talking about. I did not try it and I
> did not read the code, but I think it makes things much more
> complicated. See also the discussion about mixing package managers
> between Gentoo and Sabayon. I do not want these problems.

But your proposal adds the complexity that would bring these problems, 
whether you want them or not.  Either way, the technical problems can be 
worked thru, it's simply a matter of coding for the most part, after all, 
but the problems are there and must be dealt with either way.  You prefer 
to have Gentoo do it as a whole.  I say no, leave Gentoo well enough 
alone in that regard, and let individual projects, either as Gentoo-based 
independent distributions, or at the application (either PM or switcher 
level) deal with the problems.  After all, Gentoo's a big organization 
and getting all of it together addressing a problem is a VERY hard task 
as I'm sure CiaranM among others can vouch for.  A much smaller PM or 
switcher app project is by definition of the smaller size, also much 
easier to steer and to get things actually done in.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: better support for binary packages

[lxnay]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]



On Tue, May 26, 2009 at 9:05 AM, Philipp Riegger <lists@anderedomain.de> wrote:
> Hi Duncan,
>
>
> And all this layer thing Fabio was talking about. I did not try it and I
> did not read the code, but I think it makes things much more
> complicated. See also the discussion about mixing package managers
> between Gentoo and Sabayon. I do not want these problems.
>

incorrect. Give it a spin ;)
Problems we have were *only* related to Portage world file handling, fixed some time ago. I am sorry to say that the issue reported here doesn't seem to be valid.
Of course, if you mix both, you need to pay attention to not change USE flags (for eg.) that trigger libraries compilation, but that's a known binary-world problem.

I agree with you that there could be some more room for improvements here and there (especially in kernel module ebuilds), but with EAPI=2 we're going in the right direction.

> Philipp
>
>
>



-- 
Fabio Erculiani


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 270 bytes --]

Re: [gentoo-dev] Re: better support for binary packages

[Philipp Riegger]

On Tue, 2009-05-26 at 11:27 +0200, lxnay@sabayonlinux.org wrote:
> On Tue, May 26, 2009 at 9:05 AM, Philipp Riegger <lists@anderedomain.de> wrote:
> > And all this layer thing Fabio was talking about. I did not try it and I
> > did not read the code, but I think it makes things much more
> > complicated. See also the discussion about mixing package managers
> > between Gentoo and Sabayon. I do not want these problems.

> incorrect. Give it a spin ;)

I'll do, if i find the time.

> Problems we have were *only* related to Portage world file handling,
> fixed some time ago. I am sorry to say that the issue reported here
> doesn't seem to be valid. Of course, if you mix both, you need to pay
> attention to not change USE flags (for eg.) that trigger libraries
> compilation, but that's a known binary-world problem.

You're talking about binary-world problems here that Gentoo as a source
based distribution does not have, I assume. This is a strength of Gentoo
and I think we can keep it for binary packages by a good design. If you
emerge a package in Gentoo it gets build from source. If you use a
binary distribution, you cannot influence, what flags were used for
building the package. But with a hybrid approach (I was aming for that
one in my proposal), you would simply have the choice to either install
a binary package and be restricted, if it's not available in the way you
want it, or install it from source. And it would work together, because
you don't have 2 package managers that need to interface, talk, share,
work together, whatever, but you have 1 package manager that does it all
and can keep it consistent.

> I agree with you that there could be some more room for improvements
> here and there (especially in kernel module ebuilds), but with EAPI=2
> we're going in the right direction.

Kernel packages and kernel modules are not really of interest for me. I
would keep them as source packages. My aim is not to make thigns easier,
but to provide the user with the tools to save 8 hours of compiling
openoffice or something like that. Not a binary distribution, but some
kind of -bin packages, just packaged by Gentoo and better.

Philipp

Re: [gentoo-dev] Re: better support for binary packages

[Philipp Riegger]

On Tue, 2009-05-26 at 08:48 +0000, Duncan wrote:
> Philipp Riegger <lists@anderedomain.de> posted
> 1243321504.9661.14.camel@hspc30.informatik.uni-stuttgart.de, excerpted
> below, on  Tue, 26 May 2009 09:05:03 +0200:
> 
> > I don't see the connection between the email Fabio wrote and your
> > answer. Do you want to say, that you agree that he's doing what i
> > described and that it works the way i described it? I doubt it. If you
> > really care, could you answer my first email and state there the
> > problems you see with the approach and why you think this is making
> > Gentoo worse?
> 
> I agree that an independent approach is the way to go.  Gentoo (or 
> rather, its PMs, package managers, all three of them, of which portage is 
> only one) doesn't do binaries really well, certainly, but I really don't 
> see that changing within Gentoo itself, except at the expense of from-
> source, which it does quite well indeed.

Again, I see this as completely independent "features". Binary packages
would only support the building from source and binary packages could
never be created without ebuilds. This might even improve the quality of
the ebuilds in the tree, because they would get built from time to time,
build failures would get reported and all that.

> > But how would you make it worse? It already has the functionality to add
> > repositories for binpackages, why not improve it? Why leave it the way
> > it is?
> 
> Sure, improve it, but what you are talking about isn't a simple 
> improvement, but a massive rearchitecting.  That's not easily doable 
> without a chance of focus.  It's that change of focus that would 
> eventually kill the quality from-source support we have and that 
> continues to develop, now.

Are you sure? How code would it take to look for a binary package in a
different path than now? 1 additional line or 1 line exchanged? How much
code would it take to create this packed bit vector out of the
USE-flags? 5-10 lines? Changing portage to _also_ support the new kind
of binary packages might be done in 1 day. And this is the first step.

> >> That said, I could envision an eselect like "binary profile" switcher,
> >> that subject to settings in a config file, changes USE flags, CFLAGS,
> >> gcc- configs an appropriate gcc version, etc, changing PKGDIR
> >> appropriately as well, so one could easily enough create as many
> >> "binary profiles" as desired and as the use case dictated.
> 
> > Not sure, what the binary profile switcher really would change here.
> > What I was talking about were mostly USE-flags and packages, and I guess
> > your binary profile switcher doesn't change too much, there.
> 
> Sure it does, but incrementally.  Basically, your proposal laid out a 
> complicated tree based on metadata, a tree the package manager would have 
> to understand and support, what I'm proposing is to allow the same thing, 
> but not by adding all that complication to the package manager, but 
> rather, by using a newly created application to switch among the branches 
> of that tree on request.

Which might work, but would lead to a really strange package tree with
really big restrictions and disadvantages (no package sharing, no way of
finding out if an update is necessary,...).

> Portage (or other PM) would use its configured 
> PKGDIR and wouldn't understand the tree, but it wouldn't need to, because 
> the switcher would manage switching between the branches according to its 
> configuration.

And there the flexibility goes. If you want to emerge PHP and there is a
binary package with the same USE-flage but also cli enabled or something
like that, it would really get complicated to inform you (the user) that
a simple and probably not too important change for you might save you an
hour of compilation.

> The result would be that in a large enough deployment to have build-
> servers, the build server(s) could build a particular set of CFLAGS/USE-
> flags/gcc-version/arch/whatever, then switch to another branch and build 
> that set.  Individual package clients could simply point at the 
> appropriate tree and get most of their packages, at least the common 
> ones, that way.
> 
> Now this wouldn't directly give you the flexibility of the package name 
> changes you proposed, but it could do so indirectly, putting that 
> information in the directory tree if configured to do so.  Clients may 
> need to use the binprofile switcher as well, for individual packages they 
> chose to build outside their normal USE profile, but the process could be 
> automated once properly configured, using PM hooks as appropriate.

And as soon as you use PM hooks, you ask yourself, why you did it that
way, if not that change had made it easier, if it would have been better
to discuss the issue with portage/pkgcore/paludis developers and all
that.

Since I really like PM hooks, I'm not sure if they are really helpfull
for users. They are too cryptic. Maybe it would be wise to create some
eselect like thing for those hooks provide some basic scripts with that.


> For that matter, Gentoo already has three package managers, and binary 
> support (or the lack thereof) has been deliberately left up to the 
> package manager at this point -- it's NOT part of PMS.  It'd be equally 
> possible to either take one of the current PMs and add your enhanced 
> binary package scheme to it, or to start an independent forth package 
> manager, and have it focus on binaries rather than from-source.  (It 
> could even take the existing portage binpkgs and rename or otherwise 
> manage them as necessary, thereby avoiding the from-source side entirely, 
> if so desired.)
> 
> > Well, the last is a little bit impossible outside of Gentoo
> > (I don't want to fork the tree) and I also don't want to fork portage,
> > so the first is complicated, too.
> 
> You mention portage, but don't mention the other PMs at all.  As 
> mentioned, binary packages aren't part of PMS (the Package Management 
> Specification, the app-doc/pms package, the standard that allows PMs 
> besides portage, currently, paludis and pkgcore) at all.  Gentoo really 
> doesn't have an official binary package format at all (see PMS, Appendix 
> B, Unspecified Items), only individual package managers like portage do, 
> and at present they may conflict with each other.
> 
> That's both a good and a bad thing.  As it's not specified, you're free 
> to define it as necessary to meet your needs for any tool you devise to 
> handle binary packages.  Of course, that means that it's just that tool's 
> definition, and at least one package manager, portage, does happen to 
> have binary packages and a working format definition for them.
> 
> But that leaves you with maximum flexibility in creating whatever you 
> want, either as a separate tool (or even independent Gentoo based 
> distribution), or as an expanded PM, either forked from one of the 
> current ones, or created independently.

Now, while flexibility migth be interesting for developers and power
users, I don't think it's always the right thing for users.

Why not talk about binary packages, work on a standard and write some
draft for PMS or another document (not like "you need this", but like
the special annexes from ada for example which leave it open to be
implemented or not). Now we have sabayon. We have the portage binary
packages. How many more do we need until we realize, that it only
duplicates work?

> > And all this layer thing Fabio was talking about. I did not try it and I
> > did not read the code, but I think it makes things much more
> > complicated. See also the discussion about mixing package managers
> > between Gentoo and Sabayon. I do not want these problems.
> 
> But your proposal adds the complexity that would bring these problems, 
> whether you want them or not.  Either way, the technical problems can be 
> worked thru, it's simply a matter of coding for the most part, after all, 
> but the problems are there and must be dealt with either way.  You prefer 
> to have Gentoo do it as a whole.  I say no, leave Gentoo well enough 
> alone in that regard, and let individual projects, either as Gentoo-based 
> independent distributions, or at the application (either PM or switcher 
> level) deal with the problems.  After all, Gentoo's a big organization 
> and getting all of it together addressing a problem is a VERY hard task 
> as I'm sure CiaranM among others can vouch for.  A much smaller PM or 
> switcher app project is by definition of the smaller size, also much 
> easier to steer and to get things actually done in.

While it might be difficult, it might just be worth it.

Bit it seems to be quite an uninteresting topic, since the people most
affected by it (Gentoo developers) did not join the conversation, yet.
Maybe I should take this to gentoo-server@ and gentoo-portage@, it might
fit there.

Philipp

[gentoo-dev] Re: better support for binary packages

[Duncan]

Philipp Riegger <lists@anderedomain.de> posted
1243335643.9661.46.camel@hspc30.informatik.uni-stuttgart.de, excerpted
below, on  Tue, 26 May 2009 13:00:43 +0200:

> Bit it seems to be quite an uninteresting topic, since the people most
> affected by it (Gentoo developers) did not join the conversation, yet.
> Maybe I should take this to gentoo-server@ and gentoo-portage@, it might
> fit there.

Agreed on the participation observation and taking it elsewhere, both.  
I'd think the gentoo-portage-dev list (which I also read) would be a good 
place for hopefully more discussion, with people actually interested.  I 
still think it's likely better, at least at first, as a separate "helper" 
app, but a number of such helpers have ultimately been integrated into 
either portage itself, or into gentoolkit over time.

Also, by doing it that way rather than by trying to change Gentoo as a 
whole, you avoid the prospect of /years/ of debate that has occurred over 
GLEP55 and with it 54, which also set about to change the package naming 
conventions, in this case for ebuilds.  And given that PMS specifically 
defines binary package formats as out of its domain, I really do see that 
as the more practical approach... unless of course you /want/ to debate 
it for /years/ before anything gets done. =:^\

Then as it proves its value, it'll ultimately become the de-facto 
standard and be integrated into some future version of PMS or whatever.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: better support for binary packages

[Ben de Groot]

lxnay@sabayonlinux.org wrote:
> On Tue, May 26, 2009 at 9:05 AM, Philipp Riegger <lists@anderedomain.de>
> wrote:
>> See also the discussion about mixing package managers
>> between Gentoo and Sabayon. I do not want these problems.
> 
> incorrect. Give it a spin ;)
> Problems we have were *only* related to Portage world file handling,
> fixed some time ago. I am sorry to say that the issue reported here
> doesn't seem to be valid.
> Of course, if you mix both, you need to pay attention to not change USE
> flags (for eg.) that trigger libraries compilation, but that's a known
> binary-world problem.

So are we going to discuss this or not?
To quote your own words back at you:

> This is gentoo-dev and you are OFF TOPIC. 

> Next time, please post Sabayon specific stuff on our ML/com. channels. 

...

*If* you want to promote entropy/equo (or other sabayon work) as a
possible solution here, you should be open to discuss its shortcomings.
If not, then you should refrain from bringing it up again.

To get back on topic: I think portage's current binary support works
reasonably well, based on some experience I have with building packages
for a second, slower machine. But I can see there are shortcomings,
mainly in the described problem of storing multiple versions of a binpkg
(with different useflags etc.).

I agree with Duncan that we do not want a change of focus away from
being a source-based distribution, but then that is not a change you
would be able to "sell" to the current developers anyway. That said,
there could very well be a Gentoo project, or people contributing to
portage development, to try and improve binary package support.

-- 
Ben de Groot
Gentoo Linux developer (qt, media, lxde, desktop-misc)
Gentoo Linux Release Engineering PR liaison
______________________________________________________

Re: [gentoo-dev] Re: better support for binary packages

[lxnay]

[-- Attachment #1: Type: text/plain, Size: 2271 bytes --]



On Tue, May 26, 2009 at 7:08 PM, Ben de Groot <yngwin@gentoo.org> wrote:
> lxnay@sabayonlinux.org wrote:
>> On Tue, May 26, 2009 at 9:05 AM, Philipp Riegger <lists@anderedomain.de>
>> wrote:
>>> See also the discussion about mixing package managers
>>> between Gentoo and Sabayon. I do not want these problems.
>>
>> incorrect. Give it a spin ;)
>> Problems we have were *only* related to Portage world file handling,
>> fixed some time ago. I am sorry to say that the issue reported here
>> doesn't seem to be valid.
>> Of course, if you mix both, you need to pay attention to not change USE
>> flags (for eg.) that trigger libraries compilation, but that's a known
>> binary-world problem.
>
> So are we going to discuss this or not?
> To quote your own words back at you:
>
>> This is gentoo-dev and you are OFF TOPIC.
>
>> Next time, please post Sabayon specific stuff on our ML/com. channels.
>
> ...

My answer was completely in-topic. Even because it was just an answer.
Your rants were totally OT, OTOH.

>
> *If* you want to promote entropy/equo (or other sabayon work) as a
> possible solution here, you should be open to discuss its shortcomings.
> If not, then you should refrain from bringing it up again.

If you are just trying to annoy me, give up. It's not working ;)
I was just answering, my friend.

>
> To get back on topic: I think portage's current binary support works
> reasonably well, based on some experience I have with building packages
> for a second, slower machine. But I can see there are shortcomings,
> mainly in the described problem of storing multiple versions of a binpkg
> (with different useflags etc.).
>
> I agree with Duncan that we do not want a change of focus away from
> being a source-based distribution, but then that is not a change you
> would be able to "sell" to the current developers anyway. That said,
> there could very well be a Gentoo project, or people contributing to
> portage development, to try and improve binary package support.
>
> --
> Ben de Groot
> Gentoo Linux developer (qt, media, lxde, desktop-misc)
> Gentoo Linux Release Engineering PR liaison
> ______________________________________________________
>
>



-- 
Fabio Erculiani


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 270 bytes --]

Re: [gentoo-dev] Re: better support for binary packages

[Ben de Groot]

For the record: Fabio (lxnay) and I had a chat on IRC and cleared some
things up. He also reproduced the problem I had encountered with mixed
portage/entropy usage, and concluded that it was a bug in entropy. I'm
glad we got to a constructive point. So things turn out not to be as bad
as I was initially led to believe, and I'll give the entropy approach
the benefit of the doubt.

Cheers,
-- 
Ben de Groot
Gentoo Linux developer (qt, media, lxde, desktop-misc)
Gentoo Linux Release Engineering PR liaison
______________________________________________________