[gentoo-dev] The future of eclasses

[gentoo-dev] The future of eclasses

[Dan Armak]

Hi everyone,

About five months ago I came up with the idea of eclasses, and drobbins 
agreed for me to make a test case of all the kde ebuilds, since they are and 
were extremely similar to one another.

The test has been a success but because of big differences between inheriting 
and regular ebuilds, non-kde ebuilds have not become inheriting. They are 
also less uniform then the kde ones and are harder to generalise.

Unfortunately other developers almost never write inheriting ebuilds. The 
main complaint against eclasses as they are now, voiced by Hallski and 
others, is that they change the interface the ebuild writer has to deal with 
too radically. So developers stay away from the eclasses. Also eclasses are 
too separate from the rest of portage; at one point I had to recreate some 
basic version comparison functionality from scratch (that's scrapped now).

To remedy that, I propose to merge at least the lower levels of the ebuilds 
(e.g. base.eclass, newdepend(), etc.) into portage (e.g. ebuild.sh). What 
would be left is a high-level interface for the kde ebuilds. Ideally nothing 
but the kde*.eclass files should be left outside portage.

What might be merged includes stuff which is similar to what is already in 
portage. A quick list:

- Default functions (i.e. src_unpack calling unpack $A, src_compile calling 
make). These have existed in base.eclass for months, but recently similar 
functions have been added to ebuild.sh (the einstall stuff). Obviously this 
is needless duplication of effort.

- newdepend(), which adds its parameters to $DEPEND and $RDEPEND at the same 
time. It can also accept special parameters, for example the parameter 
"/autotools" adds autoconf, automake, make etc. to DEPEND (but not RDEPEND).

- inherit(), which is really just a stub for source eclass || die. 

- (possibly) need-qt(), need-kde(), set-kdedir(), set-qtdir(), 
qtver-from-kdever(), min-qt-ver(): these are the functions dealing with the 
location of KDE2 and 3. They tell apps where to install and against which 
kdelibs/qt to link. They are based on the make.{globals,conf} $KDE{2,3}DIR, 
$KDE{2,3}LIBSDIR settings.

What's left will then be a very simple and coherent interface for kde 
ebuilds, and will be hopefully be followed by eclasses for other things, such 
as Azarah's autotools.eclass which allows you to use autoconf 2.5 and 
automake 2.4.

I've just now committed a big cleanup of the eclasses (transparent from the 
inheriting ebuild's point of view) which eliminated several eclasses and 
brought all misc functions together in a new functions.eclass. I'll update 
the docs tomorrow.

This is all almost certainly portage v2 stuff. However I'd like everyone 
working on portage (e.g. Bevin, with whom I've spoken) to know where we 
stand, to coordinate our efforts. As always, any ideas, forecasts, 
suggestions etc. are always welcome.

-- 
Dan Armak
Gentoo Linux Developer, Desktop Team (KDE)
Matan, Israel

Re: [gentoo-dev] The future of eclasses

[Karl Trygve Kalleberg]

> This is all almost certainly portage v2 stuff. However I'd like everyone 
> working on portage (e.g. Bevin, with whom I've spoken) to know where we 
> stand, to coordinate our efforts. As always, any ideas, forecasts, 
> suggestions etc. are always welcome.

I like the idea. If its documentation becomes part of the ebuild howto,
it will get adopted fairly quickly. 

The main concern Hallski and me (and possibly others, such as John
Stalker, if memory serves) was that the current ebuild system mirrors
the manual build process so closely that external contributors have
little or no difficulty sending us ebuilds.

I think this can be solved somewhat if it is clearly documented _what_
is assumed by the various eclasses and perhaps a motivation when the
assumptions are less than crystal clear.

What plagues many non-kde ebuilds is that their build systems are fairly
non-standard. And even packages with seemingly standard build systems
quickly turn out to have both minor and major flaws (Mailman is a good
example of a horrible build system that on the surface looks nice, but 
in practice turns out to be a real bitch).

The minor nuisances of the various build systems mean that we might find
it easier to patch the generated Makefiles instead of the Makefile.in or
Makefile.am files, thus requiring an intermediate step between
configure and make (ie, patching in src_unpack is too troublesome).

Consequently a simple wrapper for src_compile will only work a certain
percentage (possibly relatively low; <= 50%) of the time, and a
full-blown manual src_compile() function must be written for the rest of
the ebuilds.

The situation for src_install() is even more dire, as it would appear
that close to all packages have trouble conforming to any kind of FHS,
and installing in a temporary directory is completely foreign.

Now, for our regular ebuild writers, the 50% saving is a big deal. For a
new submitter, it means he has to figure out which eclasses are
available (+ what they are), which criteria his package must meet to be
able to avail himself of the various eclasses, and finally test that
this is really so.

In conclusion, I am very positive about eclasses for our regular
contributors; it would possibly give us yet another advantage to our
package system that would allow us to add more ebuilds more quickly (one
could argue that quantity in itself is not a good thing, and of course I
agree ;). But if our intention is that "regular" users should easily
be able to submit ebuilds, and more easily so than writing a debian
package or an rpm spec, then we must be very careful to consider the
result eclasses have on our "regular" users.

</rant>

Kind regards,

Karl T

Re: [gentoo-dev] The future of eclasses

[John Stalker]

I agree with almost all of this, but I would like to add one additional
point.  It's not solely a matter of encouraging users to contribute
ebuilds, but also of allowing them to understand them.  I want to
understand what will happen when I type emerge foo/bar before I
actually do it.  This is what always annoyed me about rpm and
deb.  There are two ways of dealing with this.  One is to keep
everything in a familiar format, ie bash for gentoo portage or
Makefiles for FreeBSD ports, and the other is to have a stable
well-documented interface to some mysterious black box.  I would
much prefer the former wherever it is feasible, but of course I
don't have the burden of maintaining large numbers of ebuilds.

> > This is all almost certainly portage v2 stuff. However I'd like everyone 
> > working on portage (e.g. Bevin, with whom I've spoken) to know where we 
> > stand, to coordinate our efforts. As always, any ideas, forecasts, 
> > suggestions etc. are always welcome.
> 
> I like the idea. If its documentation becomes part of the ebuild howto,
> it will get adopted fairly quickly. 
> 
> The main concern Hallski and me (and possibly others, such as John
> Stalker, if memory serves) was that the current ebuild system mirrors
> the manual build process so closely that external contributors have
> little or no difficulty sending us ebuilds.
> 
> I think this can be solved somewhat if it is clearly documented _what_
> is assumed by the various eclasses and perhaps a motivation when the
> assumptions are less than crystal clear.
> 
> What plagues many non-kde ebuilds is that their build systems are fairly
> non-standard. And even packages with seemingly standard build systems
> quickly turn out to have both minor and major flaws (Mailman is a good
> example of a horrible build system that on the surface looks nice, but 
> in practice turns out to be a real bitch).
> 
> The minor nuisances of the various build systems mean that we might find
> it easier to patch the generated Makefiles instead of the Makefile.in or
> Makefile.am files, thus requiring an intermediate step between
> configure and make (ie, patching in src_unpack is too troublesome).
> 
> Consequently a simple wrapper for src_compile will only work a certain
> percentage (possibly relatively low; <= 50%) of the time, and a
> full-blown manual src_compile() function must be written for the rest of
> the ebuilds.
> 
> The situation for src_install() is even more dire, as it would appear
> that close to all packages have trouble conforming to any kind of FHS,
> and installing in a temporary directory is completely foreign.
> 
> Now, for our regular ebuild writers, the 50% saving is a big deal. For a
> new submitter, it means he has to figure out which eclasses are
> available (+ what they are), which criteria his package must meet to be
> able to avail himself of the various eclasses, and finally test that
> this is really so.
> 
In conclusion, I am very positive about eclasses for our regular
> contributors; it would possibly give us yet another advantage to our
> package system that would allow us to add more ebuilds more quickly (one
> could argue that quantity in itself is not a good thing, and of course I
> agree ;). But if our intention is that "regular" users should easily
> be able to submit ebuilds, and more easily so than writing a debian
> package or an rpm spec, then we must be very careful to consider the
> result eclasses have on our "regular" users.
> 
> </rant>
> 
> Kind regards,
> 
> Karl T
> _______________________________________________
> gentoo-dev mailing list
> gentoo-dev@gentoo.org
> http://lists.gentoo.org/mailman/listinfo/gentoo-dev

--
John Stalker
Department of Mathematics
Princeton University
(609)258-6469

Re: [gentoo-dev] The future of eclasses

[Tod M. Neidt]

Hi!

My thoughts for what their worth:

On Thu, 2002-02-07 at 13:27, Karl Trygve Kalleberg wrote:
> > This is all almost certainly portage v2 stuff. However I'd like everyone 
> > working on portage (e.g. Bevin, with whom I've spoken) to know where we 
> > stand, to coordinate our efforts. As always, any ideas, forecasts, 
> > suggestions etc. are always welcome.
> 
> I like the idea. If its documentation becomes part of the ebuild howto,
> it will get adopted fairly quickly. 
I too find the concept of eclasses "sexy." Not being a KDE user my
experience with them has been limited, but I do appreciate the
motivation behind them.  However, ....

> 
> The main concern Hallski and me (and possibly others, such as John
> Stalker, if memory serves) was that the current ebuild system mirrors
> the manual build process so closely that external contributors have
> little or no difficulty sending us ebuilds.
The structure of ebuilds being analogous to the ./configure, make, make
install manual build sequence is a tremendously valuable "feature" for
gentoo users in my opinion.  Holds to the "form follows function"
principal and is relatively transparent to a user who wants to tweak an
ebuild to suit their needs with minimal effort.  I admit that I am an
unabashed promoter of the KISS principal (one of my daily working
mantras, along with "check the connections" :)  Is their anyway that
eclasses could be "hidden" in portage so that the visible ebuilds
remains simple, i.e at least gives the appearance of being analogous to
./configure, make, make install?  As John Stalker stated in another
post, the ability to understand, or at least think you understand, with
minimal effort what is happening when you merge a particular ebuild is
attractive. 

 
> What plagues many non-kde ebuilds is that their build systems are fairly
> non-standard. And even packages with seemingly standard build systems
> quickly turn out to have both minor and major flaws (Mailman is a good
> example of a horrible build system that on the surface looks nice, but 
> in practice turns out to be a real bitch).

Many of the applications that I am interested in have roots in 30+ year
old FORTRAN and have never even had a Makefile provide in the tarball. 
Yes, they are still of value, the kind that only 30+ years of peer
review can provide.

Again, I find the eclasses concept sexy.  I am just concerned with what
might be lost, especially by the Gentoo user, in their application.

Has anyone tabulated an objective Advantages/Disadvantages analysis of
eclasses?

tod

Re: [gentoo-dev] The future of eclasses

[Dan Armak]

On Thursday 07 February 2002 19:43, you wrote:
> The structure of ebuilds being analogous to the ./configure, make, make
> install manual build sequence is a tremendously valuable "feature" for
> gentoo users in my opinion.  Holds to the "form follows function"
> principal and is relatively transparent to a user who wants to tweak an
> ebuild to suit their needs with minimal effort.  I admit that I am an
> unabashed promoter of the KISS principal (one of my daily working
> mantras, along with "check the connections" :)  Is their anyway that
> eclasses could be "hidden" in portage so that the visible ebuilds
> remains simple, i.e at least gives the appearance of being analogous to
> ./configure, make, make install?  As John Stalker stated in another
> post, the ability to understand, or at least think you understand, with
> minimal effort what is happening when you merge a particular ebuild is
> attractive.
Well, the code in the eclasses is the same exact code that wuold go into the 
ebuilds if the eclasses didn't exist. So you can read and understand it.
If you want to actually see it execute, then a) a vrey detailed log is 
generated in $WORKDIR/temp and b) you can change ebuild.sh's shell to bash -x 
and thus get info on every line executed and every action taken. It's quite 
transparent and comfortable once you're used to it.

BTW, you can export ECLASS_DEBUG_OUTPUT="on" before emerging and you'll get 
that same output to your console.

-- 
Dan Armak
Gentoo Linux Developer, Desktop Team (KDE)
Matan, Israel

Re: [gentoo-dev] The future of eclasses

[Tod M. Neidt]

On Thu, 2002-02-07 at 12:41, Dan Armak wrote:

> Well, the code in the eclasses is the same exact code that wuold go into the 
> ebuilds if the eclasses didn't exist. So you can read and understand it.
> If you want to actually see it execute, then a) a vrey detailed log is 
> generated in $WORKDIR/temp and b) you can change ebuild.sh's shell to bash -x 
> and thus get info on every line executed and every action taken. It's quite 
> transparent and comfortable once you're used to it.
> 
> BTW, you can export ECLASS_DEBUG_OUTPUT="on" before emerging and you'll get 
> that same output to your console.
Hi Dan!

Ok, ok I'm going to make it a point to merge KDE this weekend ( it will
give me an opportunity to test alsa under kde) and take a look at the
documentation you've written, as well a peruse a few ebuilds that
utilize eclasses, so that I will gain some first hand knowledge about
them.

Thank you for your patience in explaining about them.

tod

Re: [gentoo-dev] The future of eclasses

[Dan Armak]

On Thursday 07 February 2002 21:27, you wrote:
> I like the idea. If its documentation becomes part of the ebuild howto,
> it will get adopted fairly quickly.
>
> The main concern Hallski and me (and possibly others, such as John
> Stalker, if memory serves) was that the current ebuild system mirrors
> the manual build process so closely that external contributors have
> little or no difficulty sending us ebuilds.
>
> I think this can be solved somewhat if it is clearly documented _what_
> is assumed by the various eclasses and perhaps a motivation when the
> assumptions are less than crystal clear.
When parts of the eclasses are merged into portage proper, what's left will 
(I hope) be a more unified and harmonious whole. Most of what's left will be 
in kde.eclass, which is written just like a standard ebuild, so anyone can 
read it (except for the many debug output statements).

As for documentation, there's my eclass-howto (in /usr/portage/eclass/doc) 
which explains almost everything abuot eclasses and can in fact serve as a 
guide to writing new eclasses, not just inheriting ebuilds. It also has 
specific detailed instructions for writing inheriting kde ebuilds. I haven't 
updated it yet for yesterday's changes, but I shortly will.

>
> What plagues many non-kde ebuilds is that their build systems are fairly
> non-standard. And even packages with seemingly standard build systems
> quickly turn out to have both minor and major flaws (Mailman is a good
> example of a horrible build system that on the surface looks nice, but
> in practice turns out to be a real bitch).
>
> The minor nuisances of the various build systems mean that we might find
> it easier to patch the generated Makefiles instead of the Makefile.in or
> Makefile.am files, thus requiring an intermediate step between
> configure and make (ie, patching in src_unpack is too troublesome).
>
> Consequently a simple wrapper for src_compile will only work a certain
> percentage (possibly relatively low; <= 50%) of the time, and a
> full-blown manual src_compile() function must be written for the rest of
> the ebuilds.
>
> The situation for src_install() is even more dire, as it would appear
> that close to all packages have trouble conforming to any kind of FHS,
> and installing in a temporary directory is completely foreign.
I know; kde ebuilds are surprisingly uniform and standard/fhs-compliant -a 
maintainer's dream really. That's what made kde eclasses possible in the 
first place.

> Now, for our regular ebuild writers, the 50% saving is a big deal. For a
> new submitter, it means he has to figure out which eclasses are
> available (+ what they are), which criteria his package must meet to be
> able to avail himself of the various eclasses, and finally test that
> this is really so.

The following is an illustrative explanations for people who aren't very 
familiar with eclasses:
The lesson learned from eclasses is, that writing standard functions and 
overriding them in the ebuilds when they don't work still isn't efficient 
enough to justify the whole thing. Instead we must provide means to easily 
extend these functions. 

In the existing eclasses (mostly base and kde) I do it in two ways. The first 
one is function sections; quoting fom the eclass-howto (I've erased some 
parts to make this shorter):

<quote>
One rarely uses predefined functions as-is; you usually want to extend them. 
Once they have unique names (foo_src_unpack) [another eclass feature, see 
eclass-howto] it's easy to add code that executes before or after them. 
Function sections break them down and allow code to execute between any two 
sections.

The implementation is simple. Let's take as an example the src_compile() 
function from base.eclass. It looks like this:

base_src_compile() {
    ./configure || die
    make || die
}

Here is the same function, divided into sections:

base_src_compile() {

    [ -z "$1" ] && base_src_compile all

    while [ "$1" ]; do
        case $1 in
            configure)
                ./configure || die;;
            make)
                make || die;;
            all)
                base_src_compile configure make;;
        esac
    shift
    done

}

The code has been divided into two "sections": configure and make. In our 
simple example, they correspond to the two commands in the original function.

The special case all calls the same function recursively with a list of 
sections in order. The line before the block says that a call without 
parameters should be treated the same as a call with the single parameter all.

Now, in your ebuild (or eclass) that inherits from base.eclass, you get the 
stub function src_compile which calls base_src_compile without parameters. 
This makes base_src_compile execute all, that is, all its sections. You can 
leave it as-is. If you wish to extend it, you define a new src_compile and 
call base_src_compile a section at a time:

src_compile() {

    # do what i want
    base_src_compile configure
    # do something in between
    base_src_compile make
    # do something else

}

A final note: not all functions execute all their sections when called with 
all or without parameters. Some sections may be non-standard and must be 
called explicitly. The only such section right now is base_src_compile patch.
</quote>

The second way is by setting variables that the functions act on. The best 
example is that of $myconf. Many ebuilds say, in their general section:
myconf="$myconf my-parameter" # you can place a use flag? here too
Then, kde_src_compile passes $myconf to configure (after adding all the std 
parameters it knows about).

In my view, the eclass interface isn't very different from the ordinary 
ebuild one, it just encapsulates very many things, especially in the case of 
KDE where apps are stantardized. the great majority of inheriting ebuilds 
looks like this:
<headers>
. /usr/portage/eclass/inherit.eclass || die
inherit kde-base
newdepend "foo? ( bar )"
use foo && myconf="$myconf --with-foo" || myconf="$myconf --without-foo"
---

At the end of the eclass-howto I explain in detail how to write inheriting 
kde ebuilds, as well as ebuilds for apps with optional kde functionality. 
Please take a look at that; I think that (possibly with some improvements) it 
can be added to the ebuild writing howto (or a reference added).

A problem with kde ebuilds is that they *have* to use eclasses, in order for 
the kde "scheme" to work ($KDE?DIR and all that). Otherwise every kde ebuild 
would have to include some standard code that figures out where to install 
and against which kdelibs to link (it's currently in functions.eclass), so 
the obvious solution is to make a sourceable file out of it.

However, other ebuilds needn't be forced into this situation (which isn't 
strictly beneficial): I think we should avoid, at least at first, making smoe 
features only available via the eclasses. An ebuild writer should be able to 
recreate everything with minimal effort, even if it means rewriting a 
standard src_compile(). This way new ebuild authors can start out writing 
ordinary ebuilds, and later (or never if they so choose) move to inheriting 
ones. As long as there's no obligation or pressure on ebuild 
authors/developers to use eclasses, there needn't be any pressure not to.

-- 
Dan Armak
Gentoo Linux Developer, Desktop Team (KDE)
Matan, Israel