[gentoo-dev] EBUILD_FORMAT support

[gentoo-dev] EBUILD_FORMAT support

[Sven Wegener]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

Hi all!

We would like to introduce a new ebuild variable named EBUILD_FORMAT,
that tags the ebuild with a specific ebuild API version it provides or
uses. ebuilds will get an automatic dependency on a version of portage
that is required for the used API to work correctly. The mapping between
API and portage versions will be distributed with our portage tree.

There are some major ebuild API changes being discussed. These changes
need backward compatibility and the information from EBUILD_FORMAT can
be used to provide backward compatiblity in an easy way. It can be used
for the src_configure[1] support and it might as well be used to get
backward compatibility for the RDEPEND=DEPEND[2] changes.

Cheers,
Sven

[1] http://thread.gmane.org/gmane.linux.gentoo.devel/29510
[2] http://thread.gmane.org/gmane.linux.gentoo.devel/29509

-- 
Sven Wegener
Gentoo Developer
http://www.gentoo.org/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Mike Frysinger]

On Wednesday 06 July 2005 08:20 pm, Sven Wegener wrote:
> We would like to introduce a new ebuild variable named EBUILD_FORMAT,

seems like the name is much longer than it needs to be ... what's wrong with 
say 'EVER' ?
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] EBUILD_FORMAT support

[Sven Wegener]

[-- Attachment #1: Type: text/plain, Size: 481 bytes --]

On Wed, Jul 06, 2005 at 08:41:43PM -0400, Mike Frysinger wrote:
> On Wednesday 06 July 2005 08:20 pm, Sven Wegener wrote:
> > We would like to introduce a new ebuild variable named EBUILD_FORMAT,
> 
> seems like the name is much longer than it needs to be ... what's wrong with 
> say 'EVER' ?

It's fine too. EBUILD_FORMAT was just the name that fell in
#gentoo-portage once we discussed about it.

Sven

-- 
Sven Wegener
Gentoo Developer
http://www.gentoo.org/


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Nathan L. Adams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sven Wegener wrote:
> On Wed, Jul 06, 2005 at 08:41:43PM -0400, Mike Frysinger wrote:
> 
>>On Wednesday 06 July 2005 08:20 pm, Sven Wegener wrote:
>>
>>>We would like to introduce a new ebuild variable named EBUILD_FORMAT,
>>
>>seems like the name is much longer than it needs to be ... what's wrong with 
>>say 'EVER' ?
> 
> 
> It's fine too. EBUILD_FORMAT was just the name that fell in
> #gentoo-portage once we discussed about it.
> 
> Sven
> 

EVER looks like the english word 'ever'; what does it stand for? EBUILD
VERSION? If so, how about EVERSION? Since when was variable name length
a problem? Go with whatever best describes the variable and is easy to
figure out.

Nathan


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCzH772QTTR4CNEQARAlimAJ0Yh80KpXqc0yZv6Gli+KqpWaKBxQCfU6pR
2WqrKs4MfY+RCgpoFxZKD8Q=
=5nzV
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] EBUILD_FORMAT support

[Sven Wegener]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

On Wed, Jul 06, 2005 at 09:01:47PM -0400, Nathan L. Adams wrote:
> EVER looks like the english word 'ever'; what does it stand for? EBUILD
> VERSION? If so, how about EVERSION? Since when was variable name length
> a problem? Go with whatever best describes the variable and is easy to
> figure out.

Yes, ebuild version. For me all that starts with E (in combination with
Gentoo) is separated in my mind between E and the remaining part. Like:

- ebuild == e-build
- emerge == e-merge

And EVER automatically was E-VER for me, never had the idea to read it
as ever. Does that count as being addicted to Gentoo?

Sven

-- 
Sven Wegener
Gentoo Developer
http://www.gentoo.org/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Olivier Crête]

[-- Attachment #1: Type: text/plain, Size: 265 bytes --]

On Thu, 2005-07-07 at 03:09 +0200, Sven Wegener wrote:
> And EVER automatically was E-VER for me, never had the idea to read it
> as ever. Does that count as being addicted to Gentoo?

Yes it does

-- 
Olivier Crête
tester@gentoo.org
x86 Security Liaison

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Joshua Baergen]

Sven Wegener wrote:

>And EVER automatically was E-VER for me, never had the idea to read it
>as ever. Does that count as being addicted to Gentoo?
>
>Sven
>
>  
>
Under the influence at the very least...

--
Joshua Baergen
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] EBUILD_FORMAT support

[Kito]

On Jul 6, 2005, at 8:01 PM, Nathan L. Adams wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Sven Wegener wrote:
>
>> On Wed, Jul 06, 2005 at 08:41:43PM -0400, Mike Frysinger wrote:
>>
>>
>>> On Wednesday 06 July 2005 08:20 pm, Sven Wegener wrote:
>>>
>>>
>>>> We would like to introduce a new ebuild variable named  
>>>> EBUILD_FORMAT,
>>>>
>>>
>>> seems like the name is much longer than it needs to be ... what's  
>>> wrong with
>>> say 'EVER' ?
>>>
>>
>>
>> It's fine too. EBUILD_FORMAT was just the name that fell in
>> #gentoo-portage once we discussed about it.
>>
>> Sven
>>
>>
>
> EVER looks like the english word 'ever'; what does it stand for?  
> EBUILD
> VERSION? If so, how about EVERSION? Since when was variable name  
> length
> a problem? Go with whatever best describes the variable and is easy to
> figure out.

Why not follow that logic through and use something like EBUILD_API ?  
the term VERSION implies release version which of course may not be  
tied to API changes...

Kito

>
> Nathan
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.1 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
> iD8DBQFCzH772QTTR4CNEQARAlimAJ0Yh80KpXqc0yZv6Gli+KqpWaKBxQCfU6pR
> 2WqrKs4MfY+RCgpoFxZKD8Q=
> =5nzV
> -----END PGP SIGNATURE-----
> -- 
> gentoo-dev@gentoo.org mailing list
>
>

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] EBUILD_FORMAT support

[Mike Frysinger]

On Thursday 07 July 2005 12:36 am, Kito wrote:
> On Jul 6, 2005, at 8:01 PM, Nathan L. Adams wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Sven Wegener wrote:
> >> On Wed, Jul 06, 2005 at 08:41:43PM -0400, Mike Frysinger wrote:
> >>> On Wednesday 06 July 2005 08:20 pm, Sven Wegener wrote:
> >>>> We would like to introduce a new ebuild variable named
> >>>> EBUILD_FORMAT,
> >>>
> >>> seems like the name is much longer than it needs to be ... what's
> >>> wrong with
> >>> say 'EVER' ?
> >>
> >> It's fine too. EBUILD_FORMAT was just the name that fell in
> >> #gentoo-portage once we discussed about it.
> >>
> >> Sven
> >
> > EVER looks like the english word 'ever'; what does it stand for?
> > EBUILD
> > VERSION? If so, how about EVERSION? Since when was variable name
> > length
> > a problem? Go with whatever best describes the variable and is easy to
> > figure out.
>
> Why not follow that logic through and use something like EBUILD_API ?
> the term VERSION implies release version which of course may not be
> tied to API changes...

hmm that sounds even better ... EAPI ?
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] EBUILD_FORMAT support

[Ned Ludd]

On Thu, 2005-07-07 at 00:47 -0400, Mike Frysinger wrote:
> On Thursday 07 July 2005 12:36 am, Kito wrote:
> > On Jul 6, 2005, at 8:01 PM, Nathan L. Adams wrote:
> > > -----BEGIN PGP SIGNED MESSAGE-----
> > > Hash: SHA1
> > >
> > > Sven Wegener wrote:
> > >> On Wed, Jul 06, 2005 at 08:41:43PM -0400, Mike Frysinger wrote:
> > >>> On Wednesday 06 July 2005 08:20 pm, Sven Wegener wrote:
> > >>>> We would like to introduce a new ebuild variable named
> > >>>> EBUILD_FORMAT,
> > >>>
> > >>> seems like the name is much longer than it needs to be ... what's
> > >>> wrong with
> > >>> say 'EVER' ?
> > >>
> > >> It's fine too. EBUILD_FORMAT was just the name that fell in
> > >> #gentoo-portage once we discussed about it.
> > >>
> > >> Sven
> > >
> > > EVER looks like the english word 'ever'; what does it stand for?
> > > EBUILD
> > > VERSION? If so, how about EVERSION? Since when was variable name
> > > length
> > > a problem? Go with whatever best describes the variable and is easy to
> > > figure out.
> >
> > Why not follow that logic through and use something like EBUILD_API ?
> > the term VERSION implies release version which of course may not be
> > tied to API changes...
> 
> hmm that sounds even better ... EAPI ?

I would be in favor of EAPI= or an even shorter variable name.

strlen(EBUILD_FORMAT) * 19546 = 249K
strlen(EAPI) * 19546 = 77K
strlen(EV) *  19546 = 39K

Where 19546 is the number if ebuilds in the tree as.

-- 
Ned Ludd <solar@gentoo.org>

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] EBUILD_FORMAT support

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

On Thursday 07 July 2005 14:19, Ned Ludd wrote:
> I would be in favor of EAPI= or an even shorter variable name.
EAPI is probably the best name, EV makes it possible to confuse it with 
PV-like variables, referring to the versions of the package, not portage 
itself.

-- 
Diego "Flameeyes" Pettenò
Gentoo Developer - http://dev.gentoo.org/~flameeyes/
(Gentoo/FreeBSD, Video, Gentoo/AMD64, Sound, PAM)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[twofourtysix]

On 07/07/05, Ned Ludd <solar@gentoo.org> wrote:
> I would be in favor of EAPI= or an even shorter variable name.
> 
> strlen(EBUILD_FORMAT) * 19546 = 249K
> strlen(EAPI) * 19546 = 77K
> strlen(EV) *  19546 = 39K
> 
> Where 19546 is the number if ebuilds in the tree as.

If you're that interested in saving space on variable names, why not
make an eclass called, say, a.eclass, that aliases all the existing
'long' names (DESCRIPTION, SRC_URI, RDEPEND, DEPEND, IUSE, src_unpack,
...) onto 'x', 'y', 'z' and so on? Yes, another eclass would make the
tree slightly bigger, but you'd save that many times over once you
changed a few hundred ebuilds to use it.

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] EBUILD_FORMAT support

[Alec Warner]

twofourtysix wrote:

>On 07/07/05, Ned Ludd <solar@gentoo.org> wrote:
>  
>
>>I would be in favor of EAPI= or an even shorter variable name.
>>
>>strlen(EBUILD_FORMAT) * 19546 = 249K
>>strlen(EAPI) * 19546 = 77K
>>strlen(EV) *  19546 = 39K
>>
>>Where 19546 is the number if ebuilds in the tree as.
>>    
>>
>
>  
>
<sarcasm>

>If you're that interested in saving space on variable names, why not
>make an eclass called, say, a.eclass, that aliases all the existing
>'long' names (DESCRIPTION, SRC_URI, RDEPEND, DEPEND, IUSE, src_unpack,
>...) onto 'x', 'y', 'z' and so on? Yes, another eclass would make the
>tree slightly bigger, but you'd save that many times over once you
>changed a few hundred ebuilds to use it.
>
>  
>
</sarcasm>
<file:///H:/Satish/Desktop/>
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] EBUILD_FORMAT support

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

On Thu, 2005-07-07 at 08:19 -0400, Ned Ludd wrote:
> I would be in favor of EAPI= or an even shorter variable name.
> 
> strlen(EBUILD_FORMAT) * 19546 = 249K
> strlen(EAPI) * 19546 = 77K
> strlen(EV) *  19546 = 39K
> 
> Where 19546 is the number if ebuilds in the tree as.

So, the size of the tree is 500216K. This makes the percent increase in
size for the options above 0.0005%, 0.0002% and 0.0001% respectively. In
any case, nearly irrelevant to its present size.

Thanks,
Donnie

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Maurice van der Pot]

[-- Attachment #1: Type: text/plain, Size: 380 bytes --]

On Thu, Jul 07, 2005 at 10:52:06AM -0700, Donnie Berkholz intended to write:
> size for the options above 0.05%, 0.02% and 0.01% respectively. In
> any case, nearly irrelevant to its present size.

-- 
Maurice van der Pot

Gentoo Linux Developer   griffon26@gentoo.org     http://www.gentoo.org
Creator of BiteMe!       griffon26@kfk4ever.com   http://www.kfk4ever.com


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]

On Thu, Jul 07, 2005 at 10:52:06AM -0700, Donnie Berkholz wrote:
> > strlen(EBUILD_FORMAT) * 19546 = 249K
> > strlen(EAPI) * 19546 = 77K
> > strlen(EV) *  19546 = 39K
> > 
> > Where 19546 is the number if ebuilds in the tree as.
> So, the size of the tree is 500216K. This makes the percent increase in
> size for the options above 0.0005%, 0.0002% and 0.0001% respectively. In
> any case, nearly irrelevant to its present size.

Probably should count some more formatting too.
strlen(EBUILD_FORMAT="") = 16, 16 * 19546 = 306K
strlen(EAPI="") = 7, 7 * 19546 = 134K
strlen(EV="") = 5, 5 *  19546 = 96K

Don't just count the 4K blocks. Count the actual data instead (du -b).
The tree contains 95912503 bytes of data, including metadata.
Excluding metadata it is 85301916 bytes. 

It's still less than 0.4% increase in the worst case.

I like the 'EAPI' variant, as it's clearly seperate from 'EV', without
being too long.

To combat some of the increase, EAPI only needs to go into new ebuilds
that use the functionality. Old ebuilds should default to the current
syntax.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 241 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

On Thu, 2005-07-07 at 20:28 +0200, Maurice van der Pot wrote:
> On Thu, Jul 07, 2005 at 10:52:06AM -0700, Donnie Berkholz intended to write:
> > size for the options above 0.05%, 0.02% and 0.01% respectively. In
> > any case, nearly irrelevant to its present size.

Mixed up my decimal points and percent signs again, eh? I guess I only
get one.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]

On Thursday 07 July 2005 22:42, Robin H. Johnson wrote:
> On Thu, Jul 07, 2005 at 10:52:06AM -0700, Donnie Berkholz wrote:
> > > strlen(EBUILD_FORMAT) * 19546 = 249K
> > > strlen(EAPI) * 19546 = 77K
> > > strlen(EV) *  19546 = 39K
> > >
> > > Where 19546 is the number if ebuilds in the tree as.
> >
> > So, the size of the tree is 500216K. This makes the percent increase
> > in size for the options above 0.0005%, 0.0002% and 0.0001%
> > respectively. In any case, nearly irrelevant to its present size.
>
> Probably should count some more formatting too.
> strlen(EBUILD_FORMAT="") = 16, 16 * 19546 = 306K
> strlen(EAPI="") = 7, 7 * 19546 = 134K
> strlen(EV="") = 5, 5 *  19546 = 96K
>
> Don't just count the 4K blocks. Count the actual data instead (du -b).
> The tree contains 95912503 bytes of data, including metadata.
> Excluding metadata it is 85301916 bytes.
>
> It's still less than 0.4% increase in the worst case.
>
> I like the 'EAPI' variant, as it's clearly seperate from 'EV', without
> being too long.
>
> To combat some of the increase, EAPI only needs to go into new ebuilds
> that use the functionality. Old ebuilds should default to the current
> syntax.

To allow for this to work with current portage versions, perhaps it would 
be an option to introduce a new extension for .ebuild scripts that use 
it's functionality. That would allow all non-EAPI aware portage versions 
to automatically ignore ebuilds that use this.

Paul

ps. I would also suggest requiring that EAPI can be retrieved by a simple 
line by line parsing without using bash. (This allows for changing the 
parsing system)

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

On Tue, Aug 23, 2005 at 03:20:16PM +0200, Paul de Vrieze wrote:
> To allow for this to work with current portage versions, perhaps it would 
> be an option to introduce a new extension for .ebuild scripts that use 
> it's functionality. That would allow all non-EAPI aware portage versions 
> to automatically ignore ebuilds that use this.
not much for .ebuild? in the tree, personally.
Why?  Cause portage *should not* ignore those ebuilds.  If the user 
wants to merge something that is a later ebuild api then they have, at 
least portage chucks an exception that the UI can wrap into "upgrade 
portage".

With what you're proposing, we instead get bugs about portage missing 
packages.

> ps. I would also suggest requiring that EAPI can be retrieved by a simple 
> line by line parsing without using bash. (This allows for changing the 
> parsing system)
No, that's yanks EAPI setting away from eclasses.

Only time this would be required is if we move away from bash; if that 
occurs, then I'd think a new extension would be required.

As is, shifting the 'template' loaded for an ebuild can be done in 
ebd's init_environ easy enough, so no reason to add the extra 
restrictions/changes.

My 2 cents, at least ;)
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 3586 bytes --]

On Tuesday 23 August 2005 18:00, Brian Harring wrote:
> On Tue, Aug 23, 2005 at 03:20:16PM +0200, Paul de Vrieze wrote:
> > To allow for this to work with current portage versions, perhaps it
> > would be an option to introduce a new extension for .ebuild scripts
> > that use it's functionality. That would allow all non-EAPI aware
> > portage versions to automatically ignore ebuilds that use this.
>
> not much for .ebuild? in the tree, personally.
> Why?  Cause portage *should not* ignore those ebuilds.  If the user
> wants to merge something that is a later ebuild api then they have, at
> least portage chucks an exception that the UI can wrap into "upgrade
> portage".
>
> With what you're proposing, we instead get bugs about portage missing
> packages.

What I mean is compatibility with current portage versions. Current 
versions do not understand EAPI. There would be a good chance that they 
could choke on packages with all kinds of new features, even in the sync 
phase. A different extension would ensure that those portage versions 
would still work (crippled) on a new tree. Of course such an extension 
change should only be done once. Once the API versions are available this 
is not an issue.

>
> > ps. I would also suggest requiring that EAPI can be retrieved by a
> > simple line by line parsing without using bash. (This allows for
> > changing the parsing system)
>
> No, that's yanks EAPI setting away from eclasses.

If the eclasses follow similar rules that would be easilly parseable. 
(taking inherit ...) into account is easy as long as the inherit line is 
on one line of it's own. (unconditionally) These rules that would 
allready be followed out of style reasons would make various kinds of 
parsers able to parse them.

> Only time this would be required is if we move away from bash; if that
> occurs, then I'd think a new extension would be required.

It would allow to for example restrict the ebuild format such that initial 
parsing is not done by bash (but the files are still parseable by bash). 
If we perform changes I think it should be done right in the first place.

> As is, shifting the 'template' loaded for an ebuild can be done in
> ebd's init_environ easy enough, so no reason to add the extra
> restrictions/changes.

One of the issues of ebuilds is the cache/metadata stuff. Parsing an 
ebuild for basic information takes a lot of time. This can be done lots 
faster with a less featured parser (I've written one some day) that 
accepts 98% of all current ebuilds, just doesn't like dynamic features in 
the toplevel. Such a parser could be a python plugin and as such easy to 
use from python. However to ensure compatibility with a faster parser the 
EAPI variable should be there in a way that is a little more strict than 
the other variables. And such a restriction is in practice not a 
restriction.

The restriction I propose would be:
- If EAPI is defined in the ebuild it should be unconditional, on it's own
  line in the toplevel of the ebuild before any functions are defined.
  (preferably the first element after the comments and whitespace)

- If EAPI is not defined in the ebuild, but in an eclass, the inherit
  chain should be unconditional and direct. Further more in the eclass the
  above rules should be followed.

Please note that many of the conditions are allready true for current 
ebuilds, just portage can "handle" more.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] EBUILD_FORMAT support

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 10039 bytes --]

Pardon the delay, been putting this one off since it's going to be a 
fun one to address, and will be a bit long :)

On Thu, Aug 25, 2005 at 12:34:00PM +0200, Paul de Vrieze wrote:
> What I mean is compatibility with current portage versions. Current 
> versions do not understand EAPI. There would be a good chance that they 
> could choke on packages with all kinds of new features, even in the sync 
> phase. A different extension would ensure that those portage versions 
> would still work (crippled) on a new tree. Of course such an extension 
> change should only be done once. Once the API versions are available this 
> is not an issue.

General portage stance towards EAPI is unset EAPI == 0 (current stable 
ebuild format); if EAPI > then portage internal EAPI, unable to merge, 
which should be able to be detected during buildplan.

Current portage doesn't know about EAPI; boned in that respect I'll 
admit, but it's the case for all new features rolled out- three options 
for dealing with this
1) Usual method, deploy support, N months later use support.
2) tweak stable so it's aware and can complain.  Still requires 
   people to upgrade, just makes it so that they're not forced into 
   upgrading to 3.x; this is mainly a benefit for those who may don't 
   care to try the first few releases of 3.x when it hits (akin to 
   people dodging the first release or two of a gcc release).

Worth noting that one rather visibile aspect of EAPI=1 is that 
(assuming the council votes on it, and yay's it) glep33 *will* result 
in current eclasses being effectively abandoned w/in the N months 
after an EAPI capable portage is released.

Sound kind of bad, but people will have to upgrade for the 
capabilities.  If EAPI was pegged into portage/ebuilds already 
it wouldn't be an issue, issues could be detected prior.  
Unfortunately it's not, and introduction of it (and use of it) is 
going to involve a few road bumps.

Plus side, once it's in, portage *will* know if the ebuild is 
incompatible with the pythonic/bash ebuild code, and portage/the UI 
can act accordingly.

Meanwhile, the changes that are being pushed into EAPI are addition of 
configure phase (broken out from compile), elib addition, and eclass2 
support (same beast, different rules due to env save/restoration).

The potential for horkage on sync'ing isn't there due to the fact 
that's purely python side; ebuild*sh doesn't play into it.

Re: regen, issue isn't really there either; if you try and merge an 
eapi=0 on a non eapi aware portage, it works, same as it did before.
If you try to merge an eapi=1 ebuild you hit either an issue with 
inherit, or a bail immediately in src_compile, due to the fact eapi=1 
ebuilds will seperate configure out from compile (eapi=0 portage won't 
know to call it; no configure == failed compile).

That said, there also quite likely is a change coming down the pipe to 
the tree's cache; the change will shift the rsync'd metadata cache 
over to a key/val based cache.

Why oh why, yet another cache change?  Simple.  The change moves away 
from list based format to key:value pairs; in short it's a change that 
once made, means keys can be added to the cache from that point on 
without causing cache complaints on sync'ing.  Last cache breakage, I 
swear :P

EAPI addition being the next key tagged in; stable (not surprising) 
needs to be released with a version capable of reading both old and 
new format; once that's done, time for the usual "yo, upgrade people, 
something's coming down the line".  Same version that supports 
old and new cache format can also include rudimentary eapi awareness.

At least that's what I'm thinking.  It's roughly inline with the 
previous forced cache breakages, just in this case slipping in some 
extra support in the process.

Notices obviously would go out prior to moving on this also, along 
with a good chunk of waiting.


> > > ps. I would also suggest requiring that EAPI can be retrieved by a
> > > simple line by line parsing without using bash. (This allows for
> > > changing the parsing system)
> >
> > No, that yanks EAPI setting away from eclasses.
> 
> If the eclasses follow similar rules that would be easilly parseable. 
> (taking inherit ...) into account is easy as long as the inherit line is 
> on one line of it's own. (unconditionally) These rules that would 
> allready be followed out of style reasons would make various kinds of 
> parsers able to parse them.

while it's insane, people *can* use indirection (eg inherit $var) for 
inherit's as long as it's deterministic, always the same inherit call 
for that ebuild's data.  Don't see a good reason to ixnay that, which 
means we'd have to parse the whole enchilada, eclasses and the ebuild.

Effectively, raiding a single var out wouldn't fly; eclasses could 
override an ebuild's eapi setting for example, just like any other 
metadata key (imo).

A *true* format change, moving away from bash for example or moving to 
an executing design of ebuilds would require an extension change; such 
a change must imo anyways, since it's not a change of the ebuild env's 
template/hooks; either it's a fundamentally different model for 
ebuilds- either via no longer being bash based, or moving away from our 
declarative design of ebuilds.


> > Only time this would be required is if we move away from bash; if that
> > occurs, then I'd think a new extension would be required.
<inserting a comment> contradicting myself via above, above is correct </comment>
>
> It would allow to for example restrict the ebuild format such that initial 
> parsing is not done by bash (but the files are still parseable by bash). 
> If we perform changes I think it should be done right in the first place.
Elaborate please

> > As is, shifting the 'template' loaded for an ebuild can be done in
> > ebd's init_environ easy enough, so no reason to add the extra
> > restrictions/changes.
> 
> One of the issues of ebuilds is the cache/metadata stuff. Parsing an 
> ebuild for basic information takes a lot of time. This can be done lots 
> faster with a less featured parser (I've written one some day) that 
> accepts 98% of all current ebuilds, just doesn't like dynamic features in 
> the toplevel. Such a parser could be a python plugin and as such easy to 
> use from python. However to ensure compatibility with a faster parser the 
> EAPI variable should be there in a way that is a little more strict than 
> the other variables. And such a restriction is in practice not a 
> restriction.

Any parser that doesn't support full bash syntax isn't acceptable from 
where I sit; re: slow down, 2.1 is around 33% faster sourcing the 
whole tree (some cases 60% faster, some 5%, etc).  The speed up's are 
also what allow template's to be swapped, the eapi concept.

I'd note limiting the bash capabilities is a restriction that 
transcends anything EAPI should supply; changes to what's possible in 
the language (a subset of bash syntax as you're suggesting) are a 
seperate format from where I draw the line in the sand.

Mainly, limiting the syntax has the undesired affect of deviating from 
what users/devs know already; mistakes *will* occur.  QA tools can be 
written, but people are fallable; both in writing a QA tool, and 
abiding by the syntax subset allowed.


> The restriction I propose would be:
> - If EAPI is defined in the ebuild it should be unconditional, on it's own
>   line in the toplevel of the ebuild before any functions are defined.
>   (preferably the first element after the comments and whitespace)
> 
> - If EAPI is not defined in the ebuild, but in an eclass, the inherit
>   chain should be unconditional and direct. Further more in the eclass the
>   above rules should be followed.
> 
> Please note that many of the conditions are allready true for current 
> ebuilds, just portage can "handle" more.

inherit chain must be unconditional anyways.  re: eapi placement, I 
would view that as somewhat arbitrary; the question is what gain it 
would give.

I'd wonder about the parsing speed of your parser; the difference 
between parsing ebuilds and running from cache metadata is several 
orders of magnitude differant- the current cache backend flat_list 
and portage design properly corrected ought to widen the gap too.
General cache lookup is slow due to- 
A) bad call patterns, allowed by the api; N calls to get different 
   bits of metadata from a cpv, resulting in potentially N to disk set 
   of ops.
B) default cache requires opening/closing a file per cpv lookup; syscall's 
   are killer here.
C) every metadata lookup incurs 2 stats, ebuild and cache file.

Getting to the point; cache is 100x to 400x faster then sourcing for 
<=2.0.51.  Haven't tested it under 2.1, should be different due to 
cache and regen fixups/rewrites.

Back to the point, essentially, EAPI matters in two places; 
1) metadata transfer from the ebuild env into python side during 
   depends phase; has to know what to transfer key wise.
2) actual ebuild build phase executions; if it isn't the depends phase, 
   eapi being required so that the parser can swap drop in the appropriate 
   ebuild env template.

The restrictions suggested for EAPI would only make sense if eyeing 
#1, an alternative parser; no reason to drop the cache unless the 
parser is capable of hitting the same runtime performance the cache 
can hit (frankly, it's not possible from where I'm sitting although 
the gap can be narrowed).

So... the EAPI limitations, not much for due to the conclusion above.  

Interested in the parser however, since ebd is effectively a pipe 
hack so that pythonic portage can control ebuild.sh.  I (and others) 
have been after a bashlib for a while, just no one has crunched down 
and done it (easier said then done I suspect).

My 2 cents at least.
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Re: [gentoo-dev] EBUILD_FORMAT support

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 8414 bytes --]

On Friday 26 August 2005 09:35, Brian Harring wrote:
> Any parser that doesn't support full bash syntax isn't acceptable from
> where I sit; re: slow down, 2.1 is around 33% faster sourcing the
> whole tree (some cases 60% faster, some 5%, etc).  The speed up's are
> also what allow template's to be swapped, the eapi concept.

For the toplevel of the ebuilds there are many things that are not 
allowed. Basically things must be deterministic for the cache to work. I 
have built an extension that would parse 98% of current ebuilds properly, 
and much (more than 10 times) faster than the bash/ecache way. It is in 
the shape of a python module written in C. It just ignores the functions, 
so anything is allowed in there. As such the parser understands enough of 
bash to support it. Even variable substitution and inherit are supported. 
What's not supported is various kinds of uncommon substitution tricks 
that should probably not happen in the toplevel either.

Using EAPI would also allow to see something as capabilities. Say have 
portage support version 2-relaxed and version 2-strict. 2-relaxed has all 
the bash freedom and is parsed using bash. 2-strict would allow parsing 
by a faster parser module, but would limit the bash freedom. I don't say 
we have to do this, but if ebuild and eclass EAPI declarations follow a 
few very simple rules that are normally obeyed, it would be possible to 
support this thing in the future.

One of the problems I see with the current ebuild format is that it is 
impossible to do incompatible changes at all. This means that many 
features that might be desired can not be implemented. EAPI can relieve 
that. To make it easier there should be an easy way to get the EAPI of a 
package.

>
> I'd note limiting the bash capabilities is a restriction that
> transcends anything EAPI should supply; changes to what's possible in
> the language (a subset of bash syntax as you're suggesting) are a
> seperate format from where I draw the line in the sand.

What I suggest is making a policy that would make this possible in the 
future. Note that I do not wish to restrict any bash functionality in the 
various functions in the ebuild. 

> Mainly, limiting the syntax has the undesired affect of deviating from
> what users/devs know already; mistakes *will* occur.  QA tools can be
> written, but people are fallable; both in writing a QA tool, and
> abiding by the syntax subset allowed.

The QA tools would just be running the parser. If the parser chokes (which 
it doesn't easilly) then the ebuild does not conform to the correct 
syntax. It's even possible to just compare the variables returned. If 
they don't match, the format is wrong for the C parser.

>
> > The restriction I propose would be:
> > - If EAPI is defined in the ebuild it should be unconditional, on
> > it's own line in the toplevel of the ebuild before any functions are
> > defined. (preferably the first element after the comments and
> > whitespace)
> >
> > - If EAPI is not defined in the ebuild, but in an eclass, the inherit
> >   chain should be unconditional and direct. Further more in the
> > eclass the above rules should be followed.
> >
> > Please note that many of the conditions are allready true for current
> > ebuilds, just portage can "handle" more.
>
> inherit chain must be unconditional anyways.  re: eapi placement, I
> would view that as somewhat arbitrary; the question is what gain it
> would give.

The gain of putting it at the top would be that there are less chances for 
parsers to have choked on incompatible syntax. If EAPI is in the top, at 
some point incompatible syntax might be allowed, and older parsers could 
still retrieve the EAPI. Of course any syntax that works on 'egrep 
"^[ \t]*EAPI[ \t]*="' should be no problem.

>
> I'd wonder about the parsing speed of your parser; the difference
> between parsing ebuilds and running from cache metadata is several
> orders of magnitude differant- the current cache backend flat_list
> and portage design properly corrected ought to widen the gap too.
> General cache lookup is slow due to-
> A) bad call patterns, allowed by the api; N calls to get different
>    bits of metadata from a cpv, resulting in potentially N to disk set
>    of ops.
> B) default cache requires opening/closing a file per cpv lookup;
> syscall's are killer here.
> C) every metadata lookup incurs 2 stats, ebuild and cache file.

This parser was part of a stranded rewrite attempt. One of the features 
was that it regarded packages and package instances (specific files) as 
objects whose attributes would be lazilly evaluated. That means that it 
would parse if not available, lookup otherwise. The speed of "emerge -s" 
is stunning on the program as it uses a directory search which is orders 
of magnitudes faster than python doing the same thing.

> Getting to the point; cache is 100x to 400x faster then sourcing for
> <=2.0.51.  Haven't tested it under 2.1, should be different due to
> cache and regen fixups/rewrites.

Don't forget the fact that bash must be execed for normal parses, and that 
python has extremely slow string handling when not using one of the 
standard parsing modules (that work in C). To put my money where my mouth 
is, I've tarred up my code and put it on my dev space:
http://dev.gentoo.org/~pauldv/portage_native-0.1.tar.bz2

Just run make in the extracted dir. The binary created is xbuildparse, 
this is a standalone parser that takes the ebuild as argument. It will 
look for eclasses in /usr/portage/eclass.

The python module can be built with "make xbuildparse.so", and includes a 
little bit of help reachable through the normal python way.
>
> Back to the point, essentially, EAPI matters in two places;
> 1) metadata transfer from the ebuild env into python side during
>    depends phase; has to know what to transfer key wise.
> 2) actual ebuild build phase executions; if it isn't the depends phase,
>    eapi being required so that the parser can swap drop in the
> appropriate ebuild env template.

I think it also matters in actually allowing future incompatible versions 
of ebuild formats. I don't mean to say good bye to the current format, 
but when redesigning the format, we should now design it for 
extensionability.

> The restrictions suggested for EAPI would only make sense if eyeing
> #1, an alternative parser; no reason to drop the cache unless the
> parser is capable of hitting the same runtime performance the cache
> can hit (frankly, it's not possible from where I'm sitting although
> the gap can be narrowed).

You're probably right, but the time needed to parse an ebuild can be 
reduced that much that parsing will not be the issue anymore, but 
building the right tree is:

time ./xbuildparse /usr/portage/sys-libs/db/db-4.2.52_p2.ebuild 
&>/dev/null

real    0m0.054s
user    0m0.048s
sys     0m0.002s

Please note that the parser is incomplete, does have some small bugs 
(don't try it on flag-o-matic as it someway goes into an endless loop), 
and could probably do some things smarter.

> So... the EAPI limitations, not much for due to the conclusion above.
>
> Interested in the parser however, since ebd is effectively a pipe
> hack so that pythonic portage can control ebuild.sh.  I (and others)
> have been after a bashlib for a while, just no one has crunched down
> and done it (easier said then done I suspect).

See it above. It does not fully understand every bash statement around. 
And important is that it currently does not understand the "if" 
statement. This is easy to add though, just wasn't added out of "policy". 
But being that even my own ebuilds (like db) use it, it should probably 
be added.

I do believe that the parser could be made usefull for most ebuilds. This 
would however still mean a small restriction in allowed syntax. The 
parser module has basically one function which is "parse" it parses an 
ebuild, the eclasses, and returns a list of variables. Not all variables 
are substituted though, I have a python function that does this. If 
people are interested I can take a look at sanitizing my whole tree and 
providing it.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Re: [gentoo-dev] EBUILD_FORMAT support

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

> Don't forget the fact that bash must be execed for normal parses, and
> that python has extremely slow string handling when not using one of
> the standard parsing modules (that work in C). To put my money where my
> mouth is, I've tarred up my code and put it on my dev space:
> http://dev.gentoo.org/~pauldv/portage_native-0.1.tar.bz2

I've fixed up a particular issue with it (getting into an endless loop) 
and made a very simple webpage for it:
http://dev.gentoo.org/~pauldv/

ps. People please be aware that this is still alpha in the sense of not 
being complete. For better working it should probably support if 
statements properly, and at least do variable substitution. It would mean 
that the parser would have to retain a state etc.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Re: [gentoo-dev] EBUILD_FORMAT support

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 747 bytes --]

On Fri, 26 Aug 2005 14:50:52 +0200 Paul de Vrieze <pauldv@gentoo.org>
wrote:
| ps. People please be aware that this is still alpha in the sense of
| not being complete. For better working it should probably support if 
| statements properly, and at least do variable substitution. It would
| mean that the parser would have to retain a state etc.

Isn't this pretty much a waste of time if it can't handle all the code
in versionator? We're trying to move people away from ugly unreliable
manual substitutions towards readable, maintainable code using the
eclass...

-- 
Ciaran McCreesh : Gentoo Developer (Vim, Shell tools, Fluxbox, Cron)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Re: [gentoo-dev] EBUILD_FORMAT support

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On Friday 26 August 2005 16:58, Ciaran McCreesh wrote:
> On Fri, 26 Aug 2005 14:50:52 +0200 Paul de Vrieze <pauldv@gentoo.org>
>
> wrote:
> | ps. People please be aware that this is still alpha in the sense of
> | not being complete. For better working it should probably support if
> | statements properly, and at least do variable substitution. It would
> | mean that the parser would have to retain a state etc.
>
> Isn't this pretty much a waste of time if it can't handle all the code
> in versionator? We're trying to move people away from ugly unreliable
> manual substitutions towards readable, maintainable code using the
> eclass...

Hey, it said "alpha". I've just been working on having it parse even more 
eclasses. It now handles multilib.eclass, and I'll be working on making 
eutils parse fully. I need to revamp the variable substitution 
recognition a bit. I'll make sure that it handles versionator too.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Re: [gentoo-dev] EBUILD_FORMAT support

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

On Friday 26 August 2005 16:58, Ciaran McCreesh wrote:
> On Fri, 26 Aug 2005 14:50:52 +0200 Paul de Vrieze <pauldv@gentoo.org>
>
> wrote:
> | ps. People please be aware that this is still alpha in the sense of
> | not being complete. For better working it should probably support if
> | statements properly, and at least do variable substitution. It would
> | mean that the parser would have to retain a state etc.
>
> Isn't this pretty much a waste of time if it can't handle all the code
> in versionator? We're trying to move people away from ugly unreliable
> manual substitutions towards readable, maintainable code using the
> eclass...

With lack of variable substitution support I mean that it just forwards 
the variable substitutions to the calling application (output). It should 
probably also be made more aware of the variables that are allways 
extended like USE and DEPEND.

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-portage-dev] Re: [gentoo-dev] EBUILD_FORMAT support

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 1817 bytes --]

On Friday 26 August 2005 17:11, Paul de Vrieze wrote:
> On Friday 26 August 2005 16:58, Ciaran McCreesh wrote:
> > On Fri, 26 Aug 2005 14:50:52 +0200 Paul de Vrieze <pauldv@gentoo.org>
> >
> > wrote:
> > | ps. People please be aware that this is still alpha in the sense of
> > | not being complete. For better working it should probably support
> > | if statements properly, and at least do variable substitution. It
> > | would mean that the parser would have to retain a state etc.
> >
> > Isn't this pretty much a waste of time if it can't handle all the
> > code in versionator? We're trying to move people away from ugly
> > unreliable manual substitutions towards readable, maintainable code
> > using the eclass...
>
> With lack of variable substitution support I mean that it just forwards
> the variable substitutions to the calling application (output). It
> should probably also be made more aware of the variables that are
> allways extended like USE and DEPEND.

I just checked the versionator eclass though, and indeed it wouldn't 
support it. Versionator uses functions inside the variables. The parser 
does not parse functions at all beyond being able to determine their end. 
Perhaps it would be best to handle versionator specially and internalize 
the functions. While it is possible to interpret the bash functions this 
would mean full bash function duplication, make the parser more complex 
and diminish the speed of the parser.

I could even do this function mimicking in such a way that nonsupported 
functions automatically get signaled as requiring compatibility mode 
(parser is uncertain about it's results, and the old parser should be 
used).

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]