[gentoo-dev] RFD: EAPI specification in ebuilds

[gentoo-dev] RFD: EAPI specification in ebuilds

[Ulrich Mueller]

Hi all,

The way how we currently specify the EAPI in ebuilds has some
problems. For example, there is no sane way to allow usage of features
of a new bash version in a new EAPI. So we are currently stuck with
bash 3.2. Also changes of global scope behaviour, like addition of new
global scope functions (similar to "inherit") are not possible.

These flaws are outlined in GLEP 55 [1]:
| In order to get the EAPI the package manager needs to source the
| ebuild, which itself needs the EAPI in the first place. Otherwise it
| imposes a serious limitation, namely every ebuild, using any of the
| future EAPIs, will have to be source'able by old package managers
| [...]

The council has voted down GLEP 55 more than a year ago, but at the
same time requested that a solution for the mentioned issues should be
found. [2] However, there was no progress since then.

The issue arose again in bug 402167 [3] where several solutions have
been discussed. Below, I try to summarise the possible options
resulting from that discussion.


*** Proposal 1: "Parse the EAPI assignment statement" ***

This first proposal would require that the syntax of the EAPI
assignment statement in ebuilds matches a well defined regular
expression. A scan of the Portage tree shows that the statement only
occurs in the following variations (using EAPI 4 as example):

   EAPI=4
   EAPI="4"
   EAPI='4'

Sometimes this is followed by whitespace or a comment (starting with
a # sign). Also, with very few exceptions the EAPI assignment occurs
within the first few lines of the ebuild. For the vast majority of
ebuilds it is in line 5.

Written in a more formal way, appropriate for a specification:
- Ebuilds must contain at most one EAPI assignment statement.
- It must occur within the first N lines of the ebuild (N=10 and N=30
  have been suggested).
- The statement must match the following regular expression (extended
  regexp syntax):
  ^[ \t]*EAPI=(['"]?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$

Note: The first and the third point are already fulfilled by all
ebuilds in the Portage tree. The second point will require very few
ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).

The package manager would determine the EAPI by parsing the assignment
with above regular expression. A sanity check would be added. Citing
Zac Medico in [3]: "The fact that we can compare the probed EAPI to
the actual EAPI variable after the ebuild is sourced seems like a
perfect sanity check. We could easily detect inconsistencies and flag
such ebuilds as invalid, providing a reliable feedback mechanism to
ebuild developers."

This proposal comes in two variants:
1a) The change is applied retroactively for all EAPIs.
1b) It is only applied for EAPI 5 and later (which means that the
    result of the EAPI parsing would be discarded for earlier EAPIs).


*** Proposal 2: "EAPI in header comment" ***

A different approach would be to specify the EAPI in a specially
formatted comment in the ebuild's header. No syntax has been suggested
yet, but I believe that the following would work as a specification:
- The EAPI must be declared in a special comment in the first line of
  the ebuild's header, as follows:
- The first line of the ebuild must contain the word "ebuild",
  followed by whitespace, followed by the EAPI, followed by
  end-of-line or whitespace.

Again, the proposal comes in two variants:
2a) It is combined with a one time change of the file extension, like
    .ebuild -> .eb.
2b) The usual EAPI assignment statement in the ebuild is still
    required, at least for a transition period.

In the 2a case, the EAPI variable could be made read-only in bash
before sourcing the ebuild. In the 2b case, a sanity check similar to
the one mentioned above would be added.


What do you think?

(I really hope for a constructive discussion here. So, if you want
to comment that all of the above proposals suck and GLEP 55 is much
superior, then please open a new thread for it.)

Ulrich


[1] <http://www.gentoo.org/proj/en/glep/glep-0055.html#problem>
[2] <http://www.gentoo.org/proj/en/council/meeting-logs/20100823-summary.txt>
[3] <https://bugs.gentoo.org/show_bug.cgi?id=402167>

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 458 bytes --]

On Wed, 7 Mar 2012 21:41:02 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> (I really hope for a constructive discussion here. So, if you want
> to comment that all of the above proposals suck and GLEP 55 is much
> superior, then please open a new thread for it.)

I think if you want a constructive discussion, then you should ignore
the previous Council's decision and reconsider all the solutions that
have been presented.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alexis Ballier]

On Wed, 7 Mar 2012 21:41:02 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> Hi all,
> 
> The way how we currently specify the EAPI in ebuilds has some
> problems. For example, there is no sane way to allow usage of features
> of a new bash version in a new EAPI. So we are currently stuck with
> bash 3.2. Also changes of global scope behaviour, like addition of new
> global scope functions (similar to "inherit") are not possible.
> 
> These flaws are outlined in GLEP 55 [1]:
> | In order to get the EAPI the package manager needs to source the
> | ebuild, which itself needs the EAPI in the first place. Otherwise it
> | imposes a serious limitation, namely every ebuild, using any of the
> | future EAPIs, will have to be source'able by old package managers
> | [...]
> 
> The council has voted down GLEP 55 more than a year ago, but at the
> same time requested that a solution for the mentioned issues should be
> found. [2] However, there was no progress since then.
> 
> The issue arose again in bug 402167 [3] where several solutions have
> been discussed. Below, I try to summarise the possible options
> resulting from that discussion.
> 
> 
> *** Proposal 1: "Parse the EAPI assignment statement" ***
> 
> This first proposal would require that the syntax of the EAPI
> assignment statement in ebuilds matches a well defined regular
> expression. A scan of the Portage tree shows that the statement only
> occurs in the following variations (using EAPI 4 as example):
> 
>    EAPI=4
>    EAPI="4"
>    EAPI='4'
> 
> Sometimes this is followed by whitespace or a comment (starting with
> a # sign). Also, with very few exceptions the EAPI assignment occurs
> within the first few lines of the ebuild. For the vast majority of
> ebuilds it is in line 5.
> 
> Written in a more formal way, appropriate for a specification:
> - Ebuilds must contain at most one EAPI assignment statement.
> - It must occur within the first N lines of the ebuild (N=10 and N=30
>   have been suggested).
> - The statement must match the following regular expression (extended
>   regexp syntax):
>   ^[ \t]*EAPI=(['"]?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$
> 
> Note: The first and the third point are already fulfilled by all
> ebuilds in the Portage tree. The second point will require very few
> ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).
> 
> The package manager would determine the EAPI by parsing the assignment
> with above regular expression. A sanity check would be added. Citing
> Zac Medico in [3]: "The fact that we can compare the probed EAPI to
> the actual EAPI variable after the ebuild is sourced seems like a
> perfect sanity check. We could easily detect inconsistencies and flag
> such ebuilds as invalid, providing a reliable feedback mechanism to
> ebuild developers."
> 
> This proposal comes in two variants:
> 1a) The change is applied retroactively for all EAPIs.
> 1b) It is only applied for EAPI 5 and later (which means that the
>     result of the EAPI parsing would be discarded for earlier EAPIs).

+1 for the whole proposal, this seems to be solving the problem and is
the less intrusive change

As i understand it, $PM will need to try the regexp tingy on any ebuild
anyway, guess the EAPI then source the ebuild with the right sourcer
to get the real EAPI and compare it. So, for me, the proposal is
retroactive. The check whether guessed and actual EAPI are the same can
be made a warning, or just ignored, in EAPI<5 and fatal in >=5 though.

A.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[David Leverton]

On 7 March 2012 21:07, Alexis Ballier <aballier@gentoo.org> wrote:
> As i understand it, $PM will need to try the regexp tingy on any ebuild
> anyway, guess the EAPI then source the ebuild with the right sourcer
> to get the real EAPI and compare it.

Not exactly... the idea with proposal 2) is that the header comment
becomes the One True Way to set the EAPI, so it wouldn't be "guessing"
as such, and ebuilds wouldn't be allowed to change it during sourcing
any more than they can redefine PV or the like.  As mentioned, 2b)
still risks a mismatch between the header and the assignment, but
hopefully that would be temporary and the assignments could be dropped
eventually.  (And I'd suggest that the header EAPI is still considered
authoritative if present - the mismatch check should probably be a
hard error, or if not it should generate a warning and use the header
value.  There should be minimal if any risk of this changing behaviour
for any existing ebuild, since existing ebuilds almost certainly won't
match the chosen header syntax.)

As for my opinion, I would prefer 2a), or 2b) as a second choice.
IMHO it's much simpler and cleaner to come up with a strict,
well-defined header format than to try and work within (an arbitrarily
restricted version of) bash syntax.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/07/2012 03:41 PM, Ulrich Mueller wrote:
>
> *** Proposal 2: "EAPI in header comment" ***
>
> A different approach would be to specify the EAPI in a specially
> formatted comment in the ebuild's header. No syntax has been suggested
> yet, but I believe that the following would work as a specification:
> - The EAPI must be declared in a special comment in the first line of
>    the ebuild's header, as follows:
> - The first line of the ebuild must contain the word "ebuild",
>    followed by whitespace, followed by the EAPI, followed by
>    end-of-line or whitespace.
>

Someone suggested using a standard shebang the last time this came up, 
and if I remember correctly it was one of the least-disagreeable 
solutions proposed. We could of course define our own custom format, but 
I think something like,

   #!/usr/bin/eapi5

would be perfect if we could hand off the interpretation of the ebuild 
to that program. That solves the problem with new bash features, too, 
since you could point that command at a specific version.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ulrich Mueller]

>>>>> On Wed, 07 Mar 2012, Michael Orlitzky wrote:

> Someone suggested using a standard shebang the last time this came
> up, and if I remember correctly it was one of the least-disagreeable
> solutions proposed. We could of course define our own custom format,
> but I think something like,

>    #!/usr/bin/eapi5

> would be perfect if we could hand off the interpretation of the
> ebuild to that program. That solves the problem with new bash
> features, too, since you could point that command at a specific
> version.

Please note that I've formulated the specification in a way that
accounts for this:

>> - The first line of the ebuild must contain the word "ebuild",
>>   followed by whitespace, followed by the EAPI, followed by
>>   end-of-line or whitespace.

This would also match the line "#!/usr/bin/ebuild 5" (if the magic
word should be "ebuild" or "eapi" can be discussed of course).
So even if we would introduce the shebang only in a later EAPI,
previous portage versions would still recognise the new header.

Ulrich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

On Wed, 07 Mar 2012 17:14:13 -0500
Michael Orlitzky <michael@orlitzky.com> wrote:

> On 03/07/2012 03:41 PM, Ulrich Mueller wrote:
> >
> > *** Proposal 2: "EAPI in header comment" ***
> >
> > A different approach would be to specify the EAPI in a specially
> > formatted comment in the ebuild's header. No syntax has been
> > suggested yet, but I believe that the following would work as a
> > specification:
> > - The EAPI must be declared in a special comment in the first line
> > of the ebuild's header, as follows:
> > - The first line of the ebuild must contain the word "ebuild",
> >    followed by whitespace, followed by the EAPI, followed by
> >    end-of-line or whitespace.
> >
> 
> Someone suggested using a standard shebang the last time this came
> up, and if I remember correctly it was one of the least-disagreeable 
> solutions proposed. We could of course define our own custom format,
> but I think something like,
> 
>    #!/usr/bin/eapi5
> 
> would be perfect if we could hand off the interpretation of the
> ebuild to that program. That solves the problem with new bash
> features, too, since you could point that command at a specific
> version.

And what would /usr/bin/eapi5 do? Are you suggesting misusing shebang
or making ebuilds PM-centric?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/08/2012 07:03 AM, Michał Górny wrote:
>>
>> Someone suggested using a standard shebang the last time this came
>> up, and if I remember correctly it was one of the least-disagreeable
>> solutions proposed. We could of course define our own custom format,
>> but I think something like,
>>
>>     #!/usr/bin/eapi5
>>
>> would be perfect if we could hand off the interpretation of the
>> ebuild to that program. That solves the problem with new bash
>> features, too, since you could point that command at a specific
>> version.
>
> And what would /usr/bin/eapi5 do? Are you suggesting misusing shebang
> or making ebuilds PM-centric?
>

I was saying that I'd prefer a more-standard use of the shebang (if 
possible), rather than defining our own header comment syntax. Either 
way I think the second option is cleaner than regular expressions.

Right now, we're guaranteed the features of bash-3.2. I guess we 
actually use whatever is executing ebuild.sh to source them. But we need 
to interpret the ebuild file with something: we might as well put *that* 
in the shebang, since that's what it's for.

So if we were to do this with an ebuild right now, we'd add,

   #!/usr/bin/eapi4

to the header, and instead of sourcing the ebuild with whatever 
ebuild.sh is using, we would run it with 'eapi4' and pass whatever we 
need back and forth. Or maybe ebuild.sh would reload itself using 
'eapi4'. If any of that makes sense, the PMS would just need to specify 
some requirements on the shebang command.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2283 bytes --]

On Thu, 08 Mar 2012 10:56:21 -0500
Michael Orlitzky <michael@orlitzky.com> wrote:

> On 03/08/2012 07:03 AM, Michał Górny wrote:
> >>
> >> Someone suggested using a standard shebang the last time this came
> >> up, and if I remember correctly it was one of the
> >> least-disagreeable solutions proposed. We could of course define
> >> our own custom format, but I think something like,
> >>
> >>     #!/usr/bin/eapi5
> >>
> >> would be perfect if we could hand off the interpretation of the
> >> ebuild to that program. That solves the problem with new bash
> >> features, too, since you could point that command at a specific
> >> version.
> >
> > And what would /usr/bin/eapi5 do? Are you suggesting misusing
> > shebang or making ebuilds PM-centric?
> >
> 
> I was saying that I'd prefer a more-standard use of the shebang (if 
> possible), rather than defining our own header comment syntax. Either 
> way I think the second option is cleaner than regular expressions.
> 
> Right now, we're guaranteed the features of bash-3.2. I guess we 
> actually use whatever is executing ebuild.sh to source them. But we
> need to interpret the ebuild file with something: we might as well
> put *that* in the shebang, since that's what it's for.
> 
> So if we were to do this with an ebuild right now, we'd add,
> 
>    #!/usr/bin/eapi4
> 
> to the header, and instead of sourcing the ebuild with whatever 
> ebuild.sh is using, we would run it with 'eapi4' and pass whatever we 
> need back and forth. Or maybe ebuild.sh would reload itself using 
> 'eapi4'. If any of that makes sense, the PMS would just need to
> specify some requirements on the shebang command.

And something will need to provide that /usr/bin/eapi4 thing. And that
introduces new problems:

1) how are we going to support multiple package managers? will we need
to install eapi4 thing as a smart wrapper choosing the right PM?

2) what about Prefix? #!/usr/bin/env eapi4 then, or proactive updating
of shebangs in synced ebuilds? and then regenerating the whole cache
(guess how long does it take to update it),

3) what should happen if user executes ebuild? the ebuild should merge
itself? with dependencies or without?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/08/2012 12:28 PM, Michał Górny wrote:
>
> And something will need to provide that /usr/bin/eapi4 thing. And that
> introduces new problems:

I'm just parroting someone else's suggestion; I don't really know enough 
about the details to answer these properly. Not that that will stop me.


> 1) how are we going to support multiple package managers? will we need
> to install eapi4 thing as a smart wrapper choosing the right PM?
>
> 2) what about Prefix? #!/usr/bin/env eapi4 then, or proactive updating
> of shebangs in synced ebuilds? and then regenerating the whole cache
> (guess how long does it take to update it),

Wouldn't #!/use/bin/env eapi4 handle both (1) and (2)? You might have to 
eselect package-manager or something first if you want to use two PMs at 
once.


> 3) what should happen if user executes ebuild? the ebuild should merge
> itself? with dependencies or without?

If a user marks the ebuild executable and does ./foo-x.y.ebuild, the 
eapi4 wrapper can decide what to do. Nothing at all, print larry the 
cow, or crash (what we do now) are all fine with me =)

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

On Thu, 08 Mar 2012 12:48:51 -0500
Michael Orlitzky <michael@orlitzky.com> wrote:
> On 03/08/2012 12:28 PM, Michał Górny wrote:
> > And something will need to provide that /usr/bin/eapi4 thing. And
> > that introduces new problems:
> 
> I'm just parroting someone else's suggestion; I don't really know
> enough about the details to answer these properly. Not that that will
> stop me.

It probably should. Although in the early days the model for ebuilds
was that they were scripts that were "executed", nowadays there's so
much support required that it's better to think of ebuilds as being
data. If you did have a /usr/bin/eapi5, it would have to be implemented
as something that invoked the package manager, not as a direct
interpreter.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/08/2012 12:53 PM, Ciaran McCreesh wrote:
> On Thu, 08 Mar 2012 12:48:51 -0500
> Michael Orlitzky<michael@orlitzky.com>  wrote:
>> On 03/08/2012 12:28 PM, Michał Górny wrote:
>>> And something will need to provide that /usr/bin/eapi4 thing. And
>>> that introduces new problems:
>>
>> I'm just parroting someone else's suggestion; I don't really know
>> enough about the details to answer these properly. Not that that will
>> stop me.
>
> It probably should. Although in the early days the model for ebuilds
> was that they were scripts that were "executed", nowadays there's so
> much support required that it's better to think of ebuilds as being
> data. If you did have a /usr/bin/eapi5, it would have to be implemented
> as something that invoked the package manager, not as a direct
> interpreter.

Fair enough, but aren't you arguing the opposite point with Zac? If 
ebuilds are data, fine, we write EAPI=4 somewhere and be done with it. 
Anything not having that format is out-of-spec.

If they're code, they're code, and we need to execute them somehow. And 
the reason for the proposal in the first place was that the way we do it 
now ain't so great, eh?

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1953 bytes --]

On Thu, 08 Mar 2012 13:37:09 -0500
Michael Orlitzky <michael@orlitzky.com> wrote:
> > It probably should. Although in the early days the model for ebuilds
> > was that they were scripts that were "executed", nowadays there's so
> > much support required that it's better to think of ebuilds as being
> > data. If you did have a /usr/bin/eapi5, it would have to be
> > implemented as something that invoked the package manager, not as a
> > direct interpreter.
> 
> Fair enough, but aren't you arguing the opposite point with Zac? If
> ebuilds are data, fine, we write EAPI=4 somewhere and be done with
> it. Anything not having that format is out-of-spec.

The problem is that right now there's no way to determine the format of
the data until you already know the format of the data. We hack around
this by not allowing "drastic" format changes, where "drastic" includes
"using things in newer versions of bash" and "not adding new global
scope commands".

The question under discussion is whether we a) keep "what format the
data is in" as being part of the data, but impose some strange and
arbitrary conditions on it, b) make a one-time change to have some kind
of 'header' inside the file describing its format that isn't really part
of the data itself, or c) admit that GLEP 55 already solved the problem
and we might as well just fix the issue properly once and for all, even
if GLEP 55's author is considered by some to be one of Satan's little
minions.

> If they're code, they're code, and we need to execute them somehow.

The notion of "execute them somehow" that's used doesn't fit in with
the #! interpreter model. You aren't executing ebuilds via an
interpreter. You're performing an action that involves using the data
and code in an ebuild multiple times and in multiple different ways,
and that may also involve doing the same to an installed package that
is being replaced.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/08/2012 01:48 PM, Ciaran McCreesh wrote:
>
>> If they're code, they're code, and we need to execute them somehow.
>
> The notion of "execute them somehow" that's used doesn't fit in with
> the #! interpreter model. You aren't executing ebuilds via an
> interpreter. You're performing an action that involves using the data
> and code in an ebuild multiple times and in multiple different ways,
> and that may also involve doing the same to an installed package that
> is being replaced.
>

I do understand that; but the fact that the data are computed in an ugly 
turing-complete language complicates things.

Did someone already propose replacing EAPI=foo with a function call akin 
to inherit?

   eapi 4
   inherit whatever
   ...

the call to eapi() would then set $EAPI accordingly. If the ebuild is 
being executed directly, it could exit $EAPI; otherwise, it would 
continue normally. That would give us an interface to the variable, and 
we wouldn't need to know the EAPI ahead of time to do it as long as it's 
the first function called in the ebuild.

This is of course isomorphic to requiring a specific EAPI=4 format, but 
does allow you to do stupid things like x=`seq 4 4`; eapi $x; if you want.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alec Warner]

On Thu, Mar 8, 2012 at 1:35 PM, Michael Orlitzky <michael@orlitzky.com> wrote:
> On 03/08/2012 01:48 PM, Ciaran McCreesh wrote:
>>
>>
>>> If they're code, they're code, and we need to execute them somehow.
>>
>>
>> The notion of "execute them somehow" that's used doesn't fit in with
>> the #! interpreter model. You aren't executing ebuilds via an
>> interpreter. You're performing an action that involves using the data
>> and code in an ebuild multiple times and in multiple different ways,
>> and that may also involve doing the same to an installed package that
>> is being replaced.
>>
>
> I do understand that; but the fact that the data are computed in an ugly
> turing-complete language complicates things.
>
> Did someone already propose replacing EAPI=foo with a function call akin to
> inherit?

the eapi function doesn't exist in earlier eapis, so you end up doing
the rename thing. Otherwise old package managers barf on the new
syntax. I still think going with a comment is better than this.

>
>  eapi 4
>  inherit whatever
>  ...
>
> the call to eapi() would then set $EAPI accordingly. If the ebuild is being
> executed directly, it could exit $EAPI; otherwise, it would continue
> normally. That would give us an interface to the variable, and we wouldn't
> need to know the EAPI ahead of time to do it as long as it's the first
> function called in the ebuild.
>
> This is of course isomorphic to requiring a specific EAPI=4 format, but does
> allow you to do stupid things like x=`seq 4 4`; eapi $x; if you want.
>

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/08/2012 06:31 PM, Alec Warner wrote:
>>
>> Did someone already propose replacing EAPI=foo with a function call akin to
>> inherit?
>
> the eapi function doesn't exist in earlier eapis, so you end up doing
> the rename thing. Otherwise old package managers barf on the new
> syntax. I still think going with a comment is better than this.
>

There is an apparent workaround:

   EAPI=5
   eapi 5

We could do that for however long it takes PMs to adopt the 'eapi' 
function. It requires one redundant line in the ebuild for a while, like 
the comment solution, but then ultimately we're left with standard bash 
instead of a magic comment.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1540 bytes --]

On Thu, 08 Mar 2012 16:35:14 -0500
Michael Orlitzky <michael@orlitzky.com> wrote:

> On 03/08/2012 01:48 PM, Ciaran McCreesh wrote:
> >
> >> If they're code, they're code, and we need to execute them somehow.
> >
> > The notion of "execute them somehow" that's used doesn't fit in with
> > the #! interpreter model. You aren't executing ebuilds via an
> > interpreter. You're performing an action that involves using the
> > data and code in an ebuild multiple times and in multiple different
> > ways, and that may also involve doing the same to an installed
> > package that is being replaced.
> >
> 
> I do understand that; but the fact that the data are computed in an
> ugly turing-complete language complicates things.
> 
> Did someone already propose replacing EAPI=foo with a function call
> akin to inherit?
> 
>    eapi 4
>    inherit whatever
>    ...
> 
> the call to eapi() would then set $EAPI accordingly. If the ebuild is 
> being executed directly, it could exit $EAPI; otherwise, it would 
> continue normally. That would give us an interface to the variable,
> and we wouldn't need to know the EAPI ahead of time to do it as long
> as it's the first function called in the ebuild.
> 
> This is of course isomorphic to requiring a specific EAPI=4 format,
> but does allow you to do stupid things like x=`seq 4 4`; eapi $x; if
> you want.

What advantage does it give us? We still can't change ebuild syntax in
global scope because bash will barf.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/09/2012 12:04 AM, Michał Górny wrote:
>>
>> This is of course isomorphic to requiring a specific EAPI=4 format,
>> but does allow you to do stupid things like x=`seq 4 4`; eapi $x; if
>> you want.
>
> What advantage does it give us? We still can't change ebuild syntax in
> global scope because bash will barf.
>

Not in EAPI=5, no, but once all PMs are using the eapi function we could.

The function can do any crazy thing you want. Right now, we need to 
source the entire ebuild to get at its environment. Before we source it 
for real, we just want to know the value of $EAPI. Since eapi() will be 
the first function called, it can be our interface to this variable. 
Here's a stupid but hopefully clear example:

   $ cat test.ebuild
   eapi 4
   HOMEPAGE="http://www.example.com/"
   echo "test.ebuild, current EAPI=${EAPI}"


   $ cat test-sourcer.sh
   #!/bin/bash

   function eapi() {
       if [ "$TELL_ME_YOUR_EAPI" == 1 ]; then
           exit $1
       fi
       export EAPI=$1
   }

   export TELL_ME_YOUR_EAPI=1
   `eval 'source test.ebuild'`
   echo "Found EAPI: $?"

   export TELL_ME_YOUR_EAPI=0
   source test.ebuild


This sources it once, which short-circuits at the eapi() call returning 
'4'. Then we source it a second time with EAPI=4, and see that it makes 
it past the call to eapi() where any incompatible features would be.

   $ ./test-sourcer.sh
   Found EAPI: 4
   test.ebuild, current EAPI=4

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/08/2012 09:35 PM, Michael Orlitzky wrote:
> The function can do any crazy thing you want.

We don't need a function. We need to know the EAPI before we source the
ebuild, and a function doesn't give us that.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/09/12 00:51, Zac Medico wrote:
> On 03/08/2012 09:35 PM, Michael Orlitzky wrote:
>> The function can do any crazy thing you want.
> 
> We don't need a function. We need to know the EAPI before we source the
> ebuild, and a function doesn't give us that.

Surely we can source one or two lines of the ebuild safely, like the
example shows?

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 06:42 AM, Michael Orlitzky wrote:
> On 03/09/12 00:51, Zac Medico wrote:
>> On 03/08/2012 09:35 PM, Michael Orlitzky wrote:
>>> The function can do any crazy thing you want.
>>
>> We don't need a function. We need to know the EAPI before we source the
>> ebuild, and a function doesn't give us that.
> 
> Surely we can source one or two lines of the ebuild safely, like the
> example shows?

Why would we though, when sourcing is a relatively costly operation, and
there are much more efficient ways to get the EAPI?

Also, you've got a chicken and egg problem, since you're assuming that
your eapi() function is going to be available in the global environment
when the ebuild is sourced. The whole point of knowing the EAPI in
advance is so that we will be able to make changes to the global
environment, such as add, remove, or change the behavior of _any_ function.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/09/12 10:05, Zac Medico wrote:
>>
>> Surely we can source one or two lines of the ebuild safely, like the
>> example shows?
> 
> Why would we though, when sourcing is a relatively costly operation, and
> there are much more efficient ways to get the EAPI?

There do not seem to be any that people agree on =)

I mean, it's /easy/ to get the EAPI. There's a list of ways on the GLEP,
but all of them have trade-offs.

The header comment seems to be the preference in this thread. The
advantage that the eapi function has over a comment is that it's not
magic -- it's just normal bash syntax. So we've addressed that issue at
a small performance cost (we're really only sourcing the ebuild up to
'exit').

Whether or not that trade-off is worth it, who knows. I would prefer to
make it work before we make it fast; but new ideas can't hurt
considering nobody has agreed on an old one.


> Also, you've got a chicken and egg problem, since you're assuming that
> your eapi() function is going to be available in the global environment
> when the ebuild is sourced. The whole point of knowing the EAPI in
> advance is so that we will be able to make changes to the global
> environment, such as add, remove, or change the behavior of _any_ function.

Nah, at the top of the ebuild, you would have,

 EAPI=5
 eapi 5

until package managers catch up, just like we would have to do with the
comment. Once they've caught up, we can drop the EAPI= assignment, and
begin using new features after the call to 'eapi'.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 07:21 AM, Michael Orlitzky wrote:
> The advantage that the eapi function has over a comment is that it's not
> magic -- it's just normal bash syntax. So we've addressed that issue at
> a small performance cost (we're really only sourcing the ebuild up to
> 'exit').

Also consider the case where a user syncs after not having updated for a
couple of months, and the tree contains some ebuilds with EAPIs that are
not supported by the currently installed package manager.

In this case, when resolving dependencies and filtering ebuilds based on
whether or not their EAPI is supported, spawning bash once per ebuild is
much more costly than the alternatives.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alexis Ballier]

On Fri, 09 Mar 2012 07:41:09 -0800
Zac Medico <zmedico@gentoo.org> wrote:

> On 03/09/2012 07:21 AM, Michael Orlitzky wrote:
> > The advantage that the eapi function has over a comment is that
> > it's not magic -- it's just normal bash syntax. So we've addressed
> > that issue at a small performance cost (we're really only sourcing
> > the ebuild up to 'exit').
> 
> Also consider the case where a user syncs after not having updated
> for a couple of months, and the tree contains some ebuilds with EAPIs
> that are not supported by the currently installed package manager.
> 
> In this case, when resolving dependencies and filtering ebuilds based
> on whether or not their EAPI is supported, spawning bash once per
> ebuild is much more costly than the alternatives.

isnt the whole point of the proposal to get eapi without sourcing ?

so that we can use new bash features at local or global scope without
risking that people with an old bash get syntax errors trying to get
the eapi

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 07:51 AM, Alexis Ballier wrote:
> On Fri, 09 Mar 2012 07:41:09 -0800
> Zac Medico <zmedico@gentoo.org> wrote:
> 
>> On 03/09/2012 07:21 AM, Michael Orlitzky wrote:
>>> The advantage that the eapi function has over a comment is that
>>> it's not magic -- it's just normal bash syntax. So we've addressed
>>> that issue at a small performance cost (we're really only sourcing
>>> the ebuild up to 'exit').
>>
>> Also consider the case where a user syncs after not having updated
>> for a couple of months, and the tree contains some ebuilds with EAPIs
>> that are not supported by the currently installed package manager.
>>
>> In this case, when resolving dependencies and filtering ebuilds based
>> on whether or not their EAPI is supported, spawning bash once per
>> ebuild is much more costly than the alternatives.
> 
> isnt the whole point of the proposal to get eapi without sourcing ?
> 
> so that we can use new bash features at local or global scope without
> risking that people with an old bash get syntax errors trying to get
> the eapi

Right. Michael has lost sight of the goal and is moving off on a tangent.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/09/12 10:58, Zac Medico wrote:
> On 03/09/2012 07:51 AM, Alexis Ballier wrote:
>> On Fri, 09 Mar 2012 07:41:09 -0800
>> Zac Medico <zmedico@gentoo.org> wrote:
>>
>>> On 03/09/2012 07:21 AM, Michael Orlitzky wrote:
>>>> The advantage that the eapi function has over a comment is that
>>>> it's not magic -- it's just normal bash syntax. So we've addressed
>>>> that issue at a small performance cost (we're really only sourcing
>>>> the ebuild up to 'exit').
>>>
>>> Also consider the case where a user syncs after not having updated
>>> for a couple of months, and the tree contains some ebuilds with EAPIs
>>> that are not supported by the currently installed package manager.
>>>
>>> In this case, when resolving dependencies and filtering ebuilds based
>>> on whether or not their EAPI is supported, spawning bash once per
>>> ebuild is much more costly than the alternatives.
>>
>> isnt the whole point of the proposal to get eapi without sourcing ?
>>
>> so that we can use new bash features at local or global scope without
>> risking that people with an old bash get syntax errors trying to get
>> the eapi
> 
> Right. Michael has lost sight of the goal and is moving off on a tangent.

The point was to be able to get the EAPI without crashing if the ebuild
uses newer features. If you can get the EAPI without sourcing, that
obviously accomplishes the goal. But there are other goals, too, like
not limiting the syntax of the EAPI assignment. I was just trying to
think up something that addresses them all.

In any case, yeah, it would crash and burn if someone synced his tree
with an ancient version of portage. But so would the comment solution.
If you want to fix that, we either have to rename everything (and hope
we get it right this time) or reconsider GLEP 55.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 08:49 AM, Michael Orlitzky wrote:
> The point was to be able to get the EAPI without crashing if the ebuild
> uses newer features. If you can get the EAPI without sourcing, that
> obviously accomplishes the goal. But there are other goals, too, like
> not limiting the syntax of the EAPI assignment. I was just trying to
> think up something that addresses them all.

Yeah, but you compromised on what can be considered our primary goal,
which is to obtain the EAPI without sourcing the ebuild.

> In any case, yeah, it would crash and burn if someone synced his tree
> with an ancient version of portage. But so would the comment solution.

Don't forget the possibility of bash syntax extensions triggering a
syntax error [1]. The comment and bash assignment solutions are immune
to that because bash is exceedingly unlikely to break backward
compatibility in the comment and bash assignment syntax.

[1]
http://archives.gentoo.org/gentoo-dev/msg_2c1fbbb32f5bc47756303ca18055bee4.xml
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

On Fri, 09 Mar 2012 11:49:44 -0500
Michael Orlitzky <michael@orlitzky.com> wrote:
> >> isnt the whole point of the proposal to get eapi without sourcing ?
> >>
> >> so that we can use new bash features at local or global scope
> >> without risking that people with an old bash get syntax errors
> >> trying to get the eapi
> > 
> > Right. Michael has lost sight of the goal and is moving off on a
> > tangent.
> 
> The point was to be able to get the EAPI without crashing if the
> ebuild uses newer features.

No, it's not. There's more to it than that.

Some EAPIs really require defining certain environment variables, shell
options, sandbox things etc *before* the sourcing starts. It's a massive
pain in the ass to try to handle setting that kind of thing on the fly
once the sourcing has already started. Knowing the EAPI before having
to spawn a bash process isn't just about performance, it's also about
making ebuilds much less difficult to deal with.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 11:20 AM, Ciaran McCreesh wrote:
> On Fri, 09 Mar 2012 11:49:44 -0500
> Michael Orlitzky <michael@orlitzky.com> wrote:
>>>> isnt the whole point of the proposal to get eapi without sourcing ?
>>>>
>>>> so that we can use new bash features at local or global scope
>>>> without risking that people with an old bash get syntax errors
>>>> trying to get the eapi
>>>
>>> Right. Michael has lost sight of the goal and is moving off on a
>>> tangent.
>>
>> The point was to be able to get the EAPI without crashing if the
>> ebuild uses newer features.
> 
> No, it's not. There's more to it than that.
> 
> Some EAPIs really require defining certain environment variables, shell
> options, sandbox things etc *before* the sourcing starts. It's a massive
> pain in the ass to try to handle setting that kind of thing on the fly
> once the sourcing has already started. Knowing the EAPI before having
> to spawn a bash process isn't just about performance, it's also about
> making ebuilds much less difficult to deal with.

Yeah. Another way of putting it is that the requirement to spawn a bash
process and source the ebuild adds a ridiculous amount of unnecessary
complexity, in violation of the KISS principle [1].

[1] http://en.wikipedia.org/wiki/KISS_principle
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Brian Harring]

On Sat, Mar 10, 2012 at 08:06:50AM -0800, Zac Medico wrote:
> On 03/09/2012 11:20 AM, Ciaran McCreesh wrote:
> > On Fri, 09 Mar 2012 11:49:44 -0500
> > Michael Orlitzky <michael@orlitzky.com> wrote:
> >>>> isnt the whole point of the proposal to get eapi without sourcing ?
> >>>>
> >>>> so that we can use new bash features at local or global scope
> >>>> without risking that people with an old bash get syntax errors
> >>>> trying to get the eapi
> >>>
> >>> Right. Michael has lost sight of the goal and is moving off on a
> >>> tangent.
> >>
> >> The point was to be able to get the EAPI without crashing if the
> >> ebuild uses newer features.
> > 
> > No, it's not. There's more to it than that.
> > 
> > Some EAPIs really require defining certain environment variables, shell
> > options, sandbox things etc *before* the sourcing starts. It's a massive
> > pain in the ass to try to handle setting that kind of thing on the fly
> > once the sourcing has already started. Knowing the EAPI before having
> > to spawn a bash process isn't just about performance, it's also about
> > making ebuilds much less difficult to deal with.
> 
> Yeah. Another way of putting it is that the requirement to spawn a bash
> process and source the ebuild adds a ridiculous amount of unnecessary
> complexity, in violation of the KISS principle [1].

This statement is incorrect.

Even if EAPI could be parsed via some non sourcing approach, we 
*still* have to source the ebuild to get the metadata for when the 
EAPI is supported (the vast majority of usage).  That complexity is 
there one way or another- we wouldn't be trying to extract the EAPI 
from the ebuild unless the cache was invalid/missing.

Phrasing it more bluntly: you can only avoid the sourcing step if you 
can isolate that the EAPI is unsupported (which is extremely rare in 
terms of actual user experience).  For the rest of the time (well past 
the 99% mark) sourcing is the immediate step following.


Also, stop referencing wikipedia.  People know what "trivial 
objection" and "KISS" is.  Pointing at random wikipedia links when
people object is just a form of fallacious argument from authority 
( http://en.wikipedia.org/wiki/Argument_from_authority ). :P

~brian

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/11/2012 06:55 PM, Brian Harring wrote:
> On Sat, Mar 10, 2012 at 08:06:50AM -0800, Zac Medico wrote:
>> Yeah. Another way of putting it is that the requirement to spawn a bash
>> process and source the ebuild adds a ridiculous amount of unnecessary
>> complexity, in violation of the KISS principle [1].
> 
> This statement is incorrect.
> 
> Even if EAPI could be parsed via some non sourcing approach, we 
> *still* have to source the ebuild to get the metadata for when the 
> EAPI is supported (the vast majority of usage).  That complexity is 
> there one way or another- we wouldn't be trying to extract the EAPI 
> from the ebuild unless the cache was invalid/missing.

There are a couple of other cases worth considering:

1) User downloads an overlay that doesn't provide cache. We want the
package manager to give a pretty "EAPI unsupported" message, rather than
spit out some bash syntax errors.

2) You're assuming that a package manager can validate cache for an EAPI
that it doesn't necessarily support. That's a somewhat fragile
assumption, given the complexities of cache validation, which involve
verification all files that affect metadata and those files may vary
depending on the EAPI.

> Phrasing it more bluntly: you can only avoid the sourcing step if you 
> can isolate that the EAPI is unsupported (which is extremely rare in 
> terms of actual user experience).  For the rest of the time (well past 
> the 99% mark) sourcing is the immediate step following.

For the sake of being robust in all possible cases, it's just a lot
simpler if we can obtain the EAPI simply and reliably, without spawning
bash to source the ebuild.

> Also, stop referencing wikipedia.  People know what "trivial 
> objection" and "KISS" is.

You can't assume that. On this list we've got potential to have readers
and responders with lots of different backgrounds. Your insistence on
using bash to obtain the EAPI would make me wonder if you understood the
KISS principle, if I didn't know you better.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Brian Harring]

On Sun, Mar 11, 2012 at 09:08:24PM -0700, Zac Medico wrote:
> On 03/11/2012 06:55 PM, Brian Harring wrote:
> > On Sat, Mar 10, 2012 at 08:06:50AM -0800, Zac Medico wrote:
> >> Yeah. Another way of putting it is that the requirement to spawn a bash
> >> process and source the ebuild adds a ridiculous amount of unnecessary
> >> complexity, in violation of the KISS principle [1].
> > 
> > This statement is incorrect.
> > 
> > Even if EAPI could be parsed via some non sourcing approach, we 
> > *still* have to source the ebuild to get the metadata for when the 
> > EAPI is supported (the vast majority of usage).  That complexity is 
> > there one way or another- we wouldn't be trying to extract the EAPI 
> > from the ebuild unless the cache was invalid/missing.
> 
> There are a couple of other cases worth considering:
> 
> 1) User downloads an overlay that doesn't provide cache. We want the
> package manager to give a pretty "EAPI unsupported" message, rather than
> spit out some bash syntax errors.

This criticsm pretty much applies *strictly to the existing 
implementation*.  It's disenguous busting it in this fashion.

EAPI as a function explicitly gives it an out before hitting any of 
that, eliminating your entire critique.  Same goes for parsing it out 
of the ebuild, or renaming the extension.


> 2) You're assuming that a package manager can validate cache for an EAPI
> that it doesn't necessarily support.

Actually, I'm not.

> That's a somewhat fragile
> assumption, given the complexities of cache validation, which involve
> verification all files that affect metadata and those files may vary
> depending on the EAPI.

This is a fair bit of handwavey bullshit.  The same "complexities of 
cache validation" we have to by default deal with for valid/supported 
EAPIs; the case for unsupported EAPIs is actually simpler than 
for supported EAPis, and up until 7ddb7d30, was likely working fine in 
portage.  Tweaking portage to restore that support is pretty easy.

Also note that with the sole exception of g55, every implementation 
has to deal with this specific issue, even g55 isn't fully exempt from 
it since in cleansing a cache, there is the open question of pruning 
cache entries for ebuilds known, but not understood by the local PM.

Either way, the algo is as follows: PM pulls EAPI from the ebuild 
(regardless of method); PM recognizes that it can't handle it- thus 
it stores EAPI:-$EAPI, and the normal cache ebuild checksumming; 
whether it be mtime, md5 for md5-dict, or whatever form new caches 
choose to use.

For attempts to load, the cache subsystem checks that validation 
before trusting the cache entry- this is in place and has been for a 
very, very long time.  This isn't anything new.

So... the entry point is the ebuild; we have checksums for it, and a 
prefixed versions of it's EAPI stored.  The sole deal breaker here is
if we were to decide to allow eclasses to set EAPI (rather than the
current "you're not supposed to, although we don't explicitly block 
it"- which could be tightened to be disallowed at the implementation 
level for EAPI>4).

The following scenarios exist:

1) PM still doesn't support that EAPI, looks at the cache/ebuild: 
checksums are the same, thus the stored EAPI is trustable, leading to 
the PM knowing it still can't process that ebuild and masking it 
appropriately.

2) PM still doesn't support that EAPI, looks, sees that the checksums
no longer match the ebuild.  Re-pulls metadata/EAPI; if the ebuild has 
been changed to a supported EAPI, continues on its way.  If not, 
stores the checksum/negated EAPI, and masks appropriately.

3) PM now supports that EAPI, looks, sees the negated eapi and 
recognizes it as one it knowns, and forces regeneration.

All of those scenarios are there, and easy enough to handle- this 
applies for the existing EAPI implementation additionally.


Now, this isn't to say that someone can't go and try to structure new 
features explicitly to defeat the existing validations.  They could.

The question here is whether or not new features would actually 
*break* that w/out intentionally going out of their way to do so (aka, 
skip the strawman arguments, actual examples required).


> > Phrasing it more bluntly: you can only avoid the sourcing step if you 
> > can isolate that the EAPI is unsupported (which is extremely rare in 
> > terms of actual user experience).  For the rest of the time (well past 
> > the 99% mark) sourcing is the immediate step following.
> 
> For the sake of being robust in all possible cases, it's just a lot
> simpler if we can obtain the EAPI simply and reliably, without spawning
> bash to source the ebuild.

This still is arguable; any performant PM is going to have to do the 
cache dance I mentioned above, or maintain an in memory list of 
ebuilds it knows it can't handle (which if you want to talk robust, 
fails miserably if the API isn't designed for sync invalidation of 
that global list).  Complexity, oh my.

That's the kicker from where I'm sitting; you're so focused on 
avoiding bash that you ignore the complexities surrounding it that are 
unavoidable regardless.

The EAPI function route resolve this issue w/out penalizing the normal 
(aka supported) pathways, and does so at the same robustness as 
everything sans g55 (which aims to allow us to use things other 
than bash such as the mocked xml ebuild format).  Frankly, the 
function route gives us a bit more robustness- rather than allowing 
the scenario of a PM that doesn't parse seeing a version that requires 
it, via the function route doing `eapi 5 || die` has no such failure 
mode.


> > Also, stop referencing wikipedia.  People know what "trivial 
> > objection" and "KISS" is.
> 
> You can't assume that. On this list we've got potential to have readers
> and responders with lots of different backgrounds.

You might want to include links to Linux, gentoo, and bash while 
you're at it.

Ya know, just in case.


> Your insistence on
> using bash to obtain the EAPI would make me wonder if you understood the
> KISS principle, if I didn't know you better.

Zing.

What I'd like to see, is accuracy in this discussion.  Skip the 
handwavey "complexity! complexity! complexity!" crap, same for 
selective robustness definitions.  Past attempts at this discussion 
mostly failed due to people pulling crap like this and frankly it just 
pisses people off. 

Like it or not, each solution has technical pros/cons, more 
importantly, it has an impact on developers meaning what they *like* 
carries weight (rather than what you want them to do).

Shebang/parsing EAPI w/out bash, EAPI as a function, all are reliant 
on the same cache mechanism.  That's fact.  G55 is the sole exception.

The question of what mechanism to use is dependant on what people 
actually want, and how far we're willing to stretch the ebuild format 
(versus defining a new one rather than new versions of it).

For the lifespan of gentoo thus far, our format has been bash; trying 
to shift away from bash is fine, but as long as it *is* bash we should
be designing w/ that flow, including what is least likely to have a 
developer screw up.  Pragmatic reality, especially since we're still 
stuck w/ compatibility issues getting to the new mechanism.


Frankly I'm starting to think I should just write up a table of 
pros/cons, extending what Ulm has started; in particular, including 
accurate statement of the transition costs of each (something people 
seem to be ignoring- compatibility mechanisms are *not* cheap to get 
in place).

~harring

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 282 bytes --]

On Mon, 12 Mar 2012 01:36:12 -0700
Brian Harring <ferringb@gmail.com> wrote:
> Also note that with the sole exception of g55
...
> and does so at the same robustness as everything sans g55
...
> G55 is the sole exception.

Interesting pattern, huh?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/12/2012 01:36 AM, Brian Harring wrote:
> On Sun, Mar 11, 2012 at 09:08:24PM -0700, Zac Medico wrote:
>> 1) User downloads an overlay that doesn't provide cache. We want the
>> package manager to give a pretty "EAPI unsupported" message, rather than
>> spit out some bash syntax errors.
> 
> This criticsm pretty much applies *strictly to the existing 
> implementation*.  It's disenguous busting it in this fashion.
> 
> EAPI as a function explicitly gives it an out before hitting any of 
> that, eliminating your entire critique.  Same goes for parsing it out 
> of the ebuild, or renaming the extension.

You're assuming that the ebuild calls your eapi() function before it
uses any syntax that's unsupported by the user's installed version of bash.

> 1) PM still doesn't support that EAPI, looks at the cache/ebuild: 
> checksums are the same, thus the stored EAPI is trustable, leading to 
> the PM knowing it still can't process that ebuild and masking it 
> appropriately.

You're assuming that cache is provided by the repo, which is not
guaranteed, depending on the source. Even if the cache does exist, then
you're assuming it's in a format that the package manager can reliably
parse the EAPI from, even though that EAPI may not be supported. That
may or may not reliable assumption, and having a pre-defined protocol to
directly obtain the EAPI without using the cache is much more reliable.

> What I'd like to see, is accuracy in this discussion.  Skip the 
> handwavey "complexity! complexity! complexity!" crap, same for 
> selective robustness definitions.  Past attempts at this discussion 
> mostly failed due to people pulling crap like this and frankly it just 
> pisses people off. 

It's just a symptom of people not abiding by the KISS principle. When
you start talking about an approach such as the "eapi() function"
approach which introduces lots of unnecessary complexity, it naturally
makes the whole discussion more complex and hand-wavey.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

On Mon, 12 Mar 2012 09:05:26 -0700
Zac Medico <zmedico@gentoo.org> wrote:
> It's just a symptom of people not abiding by the KISS principle.

"Abiding by the KISS principle" is what got us into this mess in the
first place. EAPI as a metadata variable is too simple to allow us to
do what we want to do.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/12/2012 09:12 AM, Ciaran McCreesh wrote:
> On Mon, 12 Mar 2012 09:05:26 -0700
> Zac Medico <zmedico@gentoo.org> wrote:
>> It's just a symptom of people not abiding by the KISS principle.
> 
> "Abiding by the KISS principle" is what got us into this mess in the
> first place. EAPI as a metadata variable is too simple to allow us to
> do what we want to do.

If our goal is to obtain the EAPI before the ebuild is sourced, then
GLEP 55 is more in alignment with the KISS principle. The "EAPI as
metadata variable" approach simply wasn't designed with our current goal
in mind.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Brian Harring]

On Mon, Mar 12, 2012 at 09:05:26AM -0700, Zac Medico wrote:
> On 03/12/2012 01:36 AM, Brian Harring wrote:
> > On Sun, Mar 11, 2012 at 09:08:24PM -0700, Zac Medico wrote:
> >> 1) User downloads an overlay that doesn't provide cache. We want the
> >> package manager to give a pretty "EAPI unsupported" message, rather than
> >> spit out some bash syntax errors.
> > 
> > This criticsm pretty much applies *strictly to the existing 
> > implementation*.  It's disenguous busting it in this fashion.
> > 
> > EAPI as a function explicitly gives it an out before hitting any of 
> > that, eliminating your entire critique.  Same goes for parsing it out 
> > of the ebuild, or renaming the extension.
> 
> You're assuming that the ebuild calls your eapi() function before it
> uses any syntax that's unsupported by the user's installed version of bash.

A bit, although that's a pretty valid assumption frankly.  For current 
bash syntax, the only thing they could do that would cause issues is 
bash regex's for example- which if they have regex's prior to the eapi 
invocation, they're doing something stupid anyways.

Regardless, detecting and suppressing isn't too hard- start sourcing 
w/ `set -e`, disabling that once `eapi` has been invoked for example.  
Pretty sure people will scream "that's horrible", but it's 
surprisingly effective.


> > 1) PM still doesn't support that EAPI, looks at the cache/ebuild: 
> > checksums are the same, thus the stored EAPI is trustable, leading to 
> > the PM knowing it still can't process that ebuild and masking it 
> > appropriately.
> 
> You're assuming that cache is provided by the repo,

Sigh.  I'm making no such assumptions, nor would I; it's a stupid line 
of thought.

All of this has to function in the absence of a cache, and that's a 
core usage scenario.


> which is not
> guaranteed, depending on the source. Even if the cache does exist, then
> you're assuming it's in a format that the package manager can reliably
> parse the EAPI from, even though that EAPI may not be supported. That
> may or may not reliable assumption, and having a pre-defined protocol to
> directly obtain the EAPI without using the cache is much more reliable.

This is a nonstatement.  To deal w/ the cache (validate it) you have 
to be able to reliably pull EAPI out of the ebuild.

That's what this whole fucking discussion is about; really not sure 
why you're trying to argue this point against eapi as a function.  As 
I already laid out, it can deal w/it, same as the rest.  Importantly,
the approach can also work across the transition period preventing 
current-day PMs (using current EAPI mechanisms) from breaking when 
used against later EAPIs that were released via `eapi as a function`.


> > What I'd like to see, is accuracy in this discussion.  Skip the 
> > handwavey "complexity! complexity! complexity!" crap, same for 
> > selective robustness definitions.  Past attempts at this discussion 
> > mostly failed due to people pulling crap like this and frankly it just 
> > pisses people off. 
> 
> It's just a symptom of people not abiding by the KISS principle. When
> you start talking about an approach such as the "eapi() function"
> approach which introduces lots of unnecessary complexity, it naturally
> makes the whole discussion more complex and hand-wavey.

With respect; you're proposing we go gum up version parsing via 
shoving EAPI directly into it.  Literally, make what is already a 
complex mess, worse.  Apply some KISS to your proposal please. ;)

Just hammering the point home; compatibility *is* complex.  Claiming 
otherwise is naive.  Case in point: your proposal breaks the shit out 
of any current-day package manager that saw such a filename.

I really have a hard time reading your posts when basic issues like 
that aren't paid attention to, but you've no problems claiming 
complexity/brokenness in other proposals.

As I said, I'd like to see some accuracy; not hand wavy buzzwords.

I'd much more like to see prototypes of peoples proposals in addition 
since at least that way they would flush out the breakages in their 
proposals (potentially dropping it in the process since some of these 
are pretty half baked).

~brian

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/13/2012 07:01 PM, Brian Harring wrote:
> With respect; you're proposing we go gum up version parsing via 
> shoving EAPI directly into it.  Literally, make what is already a 
> complex mess, worse.  Apply some KISS to your proposal please. ;)
> 
> Just hammering the point home; compatibility *is* complex.  Claiming 
> otherwise is naive.  Case in point: your proposal breaks the shit out 
> of any current-day package manager that saw such a filename.

I'm not really interested in GLEP 55 or variants of it. I was just
saying that if we do go with a GLEP 55 variant, then I'd prefer one
that's similar to the "EAPI in the filename with one-time extension
change" option [1]. There are plenty of ways to do that without making
it difficult to separate the EAPI from the version part.

[1]
http://www.gentoo.org/proj/en/glep/glep-0055.html#eapi-in-the-filename-with-one-time-extension-change
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 09/03/12 10:41 AM, Zac Medico wrote:
> On 03/09/2012 07:21 AM, Michael Orlitzky wrote:
>> The advantage that the eapi function has over a comment is that
>> it's not magic -- it's just normal bash syntax. So we've
>> addressed that issue at a small performance cost (we're really
>> only sourcing the ebuild up to 'exit').
> 
> Also consider the case where a user syncs after not having updated
> for a couple of months, and the tree contains some ebuilds with
> EAPIs that are not supported by the currently installed package
> manager.


IIRC we get this already, when the EAPI isn't supported by the version
of portage installed -- upgrading really old systems won't allow an
emerge of python-2.7 due to a too-new EAPI, and python-2.7 is needed
to upgrade to the newer portage.

I don't see how the EAPI check itself failing and thereby excluding an
ebuild is much different than the specified EAPI excluding it..?
Either way, the end user is going to have issues if they don't keep
their portage up to date.



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iF4EAREIAAYFAk9aJ0QACgkQAJxUfCtlWe2tTAEA7iUgDOCaGoQhz1dXukQ/a3lY
rsdqewd2DYZWtsv+3XoA/iRVe+qf4HXdkWTchFRHlolaTJechz6AZCzKY/sNdu4w
=1e/8
-----END PGP SIGNATURE-----

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 07:52 AM, Ian Stakenvicius wrote:
> On 09/03/12 10:41 AM, Zac Medico wrote:
>> On 03/09/2012 07:21 AM, Michael Orlitzky wrote:
>>> The advantage that the eapi function has over a comment is that
>>> it's not magic -- it's just normal bash syntax. So we've
>>> addressed that issue at a small performance cost (we're really
>>> only sourcing the ebuild up to 'exit').
> 
>> Also consider the case where a user syncs after not having updated
>> for a couple of months, and the tree contains some ebuilds with
>> EAPIs that are not supported by the currently installed package
>> manager.
> 
> 
> IIRC we get this already, when the EAPI isn't supported by the version
> of portage installed -- upgrading really old systems won't allow an
> emerge of python-2.7 due to a too-new EAPI, and python-2.7 is needed
> to upgrade to the newer portage.

You're talking about a very specific (unfortunate) upgrade path
breakage. We should keep the discussion focused on more generic cases
that apply to future upgrade scenarios, rather than focusing on past
breakages (though it's obvious that we should also try to avoid such
upgrade path breakages in the future).

Assuming that there is still a viable upgrade path, the package manager
needs resolve dependencies and filter ebuilds based on whether or not
their EAPI is supported. There are basically two ways to do the EAPI
filtering:

1) Access the EAPI from the metadata cache. This assumes that the
package manager is able to validate metadata cache entries for EAPIs
that it does not support, which is a somewhat fragile assumption given
the complexities of cache validation, which involves verification of
ebuilds and additional sources of metadata which may include eclasses
and possibly other files in the future.

2) Parse the ebuild from the ebuild using a pre-defined protocol (parse
EAPI assignment, parse comment, or parse file name as in GLEP 55...).

> I don't see how the EAPI check itself failing and thereby excluding an
> ebuild is much different than the specified EAPI excluding it..?
> Either way, the end user is going to have issues if they don't keep
> their portage up to date.

They may or may not have issues. Our goal is to minimize our
vulnerability to these kinds of issues as much as possible. Being able
to obtain the ebuild EAPI without the expense of sourcing it is one
small step toward this goal.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Eray Aslan]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On Fri, Mar 09, 2012 at 08:15:11AM -0800, Zac Medico wrote:
> They may or may not have issues. Our goal is to minimize our
> vulnerability to these kinds of issues as much as possible. Being able
> to obtain the ebuild EAPI without the expense of sourcing it is one
> small step toward this goal.

EAPI is metadata and is best treated as such.  In other words, history
aside, it does not belong inside an ebuild.  Making EAPI info part of
the filename does look like a reasonable solution - similar to
seen/replied flags in the filenames in maildir directories.  Heck, even
version numbers in an ebuild filename is similar.

I don't understand why there is a strong objection to it.

But anyway, it is Friday night and I am out of here.  Have fun.

-- 
Eray Aslan <eras@gentoo.org>

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 08:33 AM, Eray Aslan wrote:
> On Fri, Mar 09, 2012 at 08:15:11AM -0800, Zac Medico wrote:
>> They may or may not have issues. Our goal is to minimize our
>> vulnerability to these kinds of issues as much as possible. Being able
>> to obtain the ebuild EAPI without the expense of sourcing it is one
>> small step toward this goal.
> 
> EAPI is metadata and is best treated as such.  In other words, history
> aside, it does not belong inside an ebuild.  Making EAPI info part of
> the filename does look like a reasonable solution - similar to
> seen/replied flags in the filenames in maildir directories.  Heck, even
> version numbers in an ebuild filename is similar.
> 
> I don't understand why there is a strong objection to it.

I have a mild preference for the "parse EAPI assignment" approach,
simply because it's the least invasive. That said, ultimately it doesn't
make much difference to me whether we parse it from inside the ebuild or
from its file name. From my perspective, arguing between these
approaches boils down to hair-splitting [1].

[1] http://en.wikipedia.org/wiki/Trivial_objections
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

On Fri, 09 Mar 2012 00:35:14 -0500
Michael Orlitzky <michael@orlitzky.com> wrote:

> On 03/09/2012 12:04 AM, Michał Górny wrote:
> >>
> >> This is of course isomorphic to requiring a specific EAPI=4 format,
> >> but does allow you to do stupid things like x=`seq 4 4`; eapi $x;
> >> if you want.
> >
> > What advantage does it give us? We still can't change ebuild syntax
> > in global scope because bash will barf.
> >
> 
> Not in EAPI=5, no, but once all PMs are using the eapi function we
> could.
> 
> The function can do any crazy thing you want. Right now, we need to 
> source the entire ebuild to get at its environment. Before we source
> it for real, we just want to know the value of $EAPI. Since eapi()
> will be the first function called, it can be our interface to this
> variable.

What if bash starts to parse the script completely and barfs at 'syntax
error' before it starts executing stuff?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/09/12 11:29, Michał Górny wrote:
> 
> What if bash starts to parse the script completely and barfs at 'syntax
> error' before it starts executing stuff?
> 

It doesn't parse the script completely, it executes line-by-line, so we
can bail out early.

This returns 1:

  exit 1

  QWE*$)@#$%IT@$KTRGV{PO#J$%$M@#$_)TVI!%K@B)$YJMV$%())@K#@)

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ulrich Mueller]

>>>>> On Fri, 09 Mar 2012, Michael Orlitzky wrote:

>> What if bash starts to parse the script completely and barfs at
>> 'syntax error' before it starts executing stuff?

> It doesn't parse the script completely, it executes line-by-line, so
> we can bail out early.

How can you tell that this behaviour won't be changed in a future bash
version?

Ulrich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/09/12 12:11, Ulrich Mueller wrote:
>>>>>> On Fri, 09 Mar 2012, Michael Orlitzky wrote:
> 
>>> What if bash starts to parse the script completely and barfs at
>>> 'syntax error' before it starts executing stuff?
> 
>> It doesn't parse the script completely, it executes line-by-line, so
>> we can bail out early.
> 
> How can you tell that this behaviour won't be changed in a future bash
> version?
> 

Who's to say that in the future my computer won't be made out of
delicious ice cream, eliminating the need for EAPIs entirely?

Chances are, this would break thousands of scripts, so we hope they
wouldn't do it. If it does happen, we either deal with it then, or don't
upgrade to that version of bash -- the same as we would do with any
other massive breaking change.

At any rate, I'm now convinced that we all want GLEP 55, but with a
different name.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 09:31 AM, Michael Orlitzky wrote:
> On 03/09/12 12:11, Ulrich Mueller wrote:
>>>>>>> On Fri, 09 Mar 2012, Michael Orlitzky wrote:
>>
>>>> What if bash starts to parse the script completely and barfs at
>>>> 'syntax error' before it starts executing stuff?
>>
>>> It doesn't parse the script completely, it executes line-by-line, so
>>> we can bail out early.
>>
>> How can you tell that this behaviour won't be changed in a future bash
>> version?
>>
> 
> Who's to say that in the future my computer won't be made out of
> delicious ice cream, eliminating the need for EAPIs entirely?
> 
> Chances are, this would break thousands of scripts, so we hope they
> wouldn't do it. If it does happen, we either deal with it then, or don't
> upgrade to that version of bash -- the same as we would do with any
> other massive breaking change.

Ulrich is talking about extensions which require a newer version of
bash. These kinds of extensions are quite common and don't cause
"massive breaking" because people simply have to upgrade bash in order
to use the new extensions, and their old scripts continue to run because
the new extensions don't interfere with backward compatibility.

Your eapi() function proposal is especially fragile in this context
because it assumes that the installed version of bash will be able to
execute a script that may target a newer version of bash. This is a
special case that is typically not a problem, although it is a major
problem under the specific conditions that your eapi() function approach
induces.

Anyway, lets focus on our main goal, which is to decide on a way to
obtain the EAPI _without_ sourcing the ebuild.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/09/12 12:47, Zac Medico wrote:
> 
> Ulrich is talking about extensions which require a newer version of
> bash. These kinds of extensions are quite common and don't cause
> "massive breaking" because people simply have to upgrade bash in order
> to use the new extensions, and their old scripts continue to run because
> the new extensions don't interfere with backward compatibility.
> 
> Your eapi() function proposal is especially fragile in this context
> because it assumes that the installed version of bash will be able to
> execute a script that may target a newer version of bash. This is a
> special case that is typically not a problem, although it is a major
> problem under the specific conditions that your eapi() function approach
> induces.

Well, you wouldn't need to execute the script targeting a newer version
of bash. You would need to source it, which involves maybe setting
EAPI=5, and then calling the 'eapi' function which will immediately exit.

The new features aren't a problem because you never get to them. (If you
try to use some new bash extension to compute your EAPI value, well, you
shouldn't have done that.)


> Anyway, lets focus on our main goal, which is to decide on a way to
> obtain the EAPI _without_ sourcing the ebuild.

Agreed. I'm forced to agree with Ciaran: can we just ignore the previous
council's decision, and reconsider the merits of GLEP 55?

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Rich Freeman]

On Fri, Mar 9, 2012 at 12:47 PM, Zac Medico <zmedico@gentoo.org> wrote:
> Anyway, lets focus on our main goal, which is to decide on a way to
> obtain the EAPI _without_ sourcing the ebuild.

Agreed.  Plus, an approach that either uses the filename or something
like a comment line is also going to be much more flexible if bash
syntax changes substantially, or if we want to support
ebuilds-in-python or some other approach.

That is the main merit I see to the shebang approach - you could
define some kind of API that actually involves executing the ebuild.
Sticking the EAPI in the filename is a little less complicated and it
gives you the same flexibility - I'm not aware of any file format that
is fussy about the content of the filename.

Even approaches like putting EAPI=5 in the file (even inside a
comment) might break if the file is meant to be generally interpreted
by some program that has rigid syntax rules.    Most scripting
languages could probably handle this somehow, but if you ever wanted
ELF ebuilds that wouldn't be so likely to fly unless you could embed
\nEAPI=ELF8\n in something near the start of the file.

Sure, I doubt we'll ever want ELF ebuilds (I'd consider
non-programatic ebuilds for more trivial packages more likely), but
the point is that we should prefer options that offer more flexibility
down the road over ones that solve the immediate need but just leave
us with the same debate two years from now.

Rich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Brian Harring]

On Fri, Mar 09, 2012 at 09:47:50AM -0800, Zac Medico wrote:
> On 03/09/2012 09:31 AM, Michael Orlitzky wrote:
> > On 03/09/12 12:11, Ulrich Mueller wrote:
> >>>>>>> On Fri, 09 Mar 2012, Michael Orlitzky wrote:
> >>
> >>>> What if bash starts to parse the script completely and barfs at
> >>>> 'syntax error' before it starts executing stuff?
> >>
> >>> It doesn't parse the script completely, it executes line-by-line, so
> >>> we can bail out early.
> >>
> >> How can you tell that this behaviour won't be changed in a future bash
> >> version?
> >>
> > 
> > Who's to say that in the future my computer won't be made out of
> > delicious ice cream, eliminating the need for EAPIs entirely?
> > 
> > Chances are, this would break thousands of scripts, so we hope they
> > wouldn't do it. If it does happen, we either deal with it then, or don't
> > upgrade to that version of bash -- the same as we would do with any
> > other massive breaking change.
> 
> Ulrich is talking about extensions which require a newer version of
> bash. These kinds of extensions are quite common and don't cause
> "massive breaking" because people simply have to upgrade bash in order
> to use the new extensions, and their old scripts continue to run because
> the new extensions don't interfere with backward compatibility.
> 
> Your eapi() function proposal is especially fragile in this context
> because it assumes that the installed version of bash will be able to
> execute a script that may target a newer version of bash. This is a
> special case that is typically not a problem, although it is a major
> problem under the specific conditions that your eapi() function approach
> induces.

Pragmatic reality, the eapi function actually would work fine.  As 
pointed out elsewhere, bash parses as it goes- which isn't going to 
change.

If someone invokes 'eapi happy-meal' and it's not supported, 
the sourcing is stopped immediately, cache gets -happy-meal for the 
EAPI, and the pm continues to ignore the ebuild till either the 
ebuilds mtime changes (which case it redoes the metadata regen) or the 
PM now supports that EAPI, and... you guessed it, redoes the metadata 
regen.  The cache behaviour is basically the same regardless of the 
EAPI mechanism.

Now, this isn't to say everyone views the function as *optimal*.  
People can argue about that as much as they'd like.

The point however is that it *would* work.  Anyone claiming 
fragility/otherwise needs to put forth actual code examples.


> Anyway, lets focus on our main goal, which is to decide on a way to
> obtain the EAPI _without_ sourcing the ebuild.

Nitpicking, but the point needs be made; this is *your* requirement.  
The requirement is to be able to deploy new globals/bash 
requirements/whatever.

There is a difference.

~brian

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Rich Freeman]

On Sun, Mar 11, 2012 at 10:03 PM, Brian Harring <ferringb@gmail.com> wrote:
> Pragmatic reality, the eapi function actually would work fine.  As
> pointed out elsewhere, bash parses as it goes- which isn't going to
> change.

Unless the ebuild isn't written in bash...

How do you source the ebuild if you don't know what to use to source
it?  How do you know what to use to source it if you don't know the
EAPI?  Right now all the existing EAPIs use bash, but there is no
reason the file couldn't be xml, or python, or just about anything
else.

If we want to allow for that kind of flexibility, then it might make
sense to go ahead and state that our convention is to stick EAPI=5 in
one of the first few lines of the ebuild, or inside a comment, but
also go a step further and state that the text "EAPI=" cannot appear
elsewhere in the ebuild (or perhaps within the first 10 lines).  Just
about any file format we might use would allow us to make "EAPI="
appear in it, but not all could guarantee that it would occur at the
start of a line, or at the start of a line immediately after a #.

In any case, I can really see the KISS value in a very rigid syntax
that is trivial to parse.  Stuff like this almost makes me wish our
ebuilds already were xml files or such, with bash embedded inside
sections.  Finding a particular tag in an xml file is trivial as the
fundamentals haven't changed in 15 years.

Rich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alec Warner]

On Sun, Mar 11, 2012 at 7:20 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Sun, Mar 11, 2012 at 10:03 PM, Brian Harring <ferringb@gmail.com> wrote:
>> Pragmatic reality, the eapi function actually would work fine.  As
>> pointed out elsewhere, bash parses as it goes- which isn't going to
>> change.
>
> Unless the ebuild isn't written in bash...

I'd opt for a different extension in that case actually.

>
> How do you source the ebuild if you don't know what to use to source
> it?  How do you know what to use to source it if you don't know the
> EAPI?  Right now all the existing EAPIs use bash, but there is no
> reason the file couldn't be xml, or python, or just about anything
> else.
>
> If we want to allow for that kind of flexibility, then it might make
> sense to go ahead and state that our convention is to stick EAPI=5 in
> one of the first few lines of the ebuild, or inside a comment, but
> also go a step further and state that the text "EAPI=" cannot appear
> elsewhere in the ebuild (or perhaps within the first 10 lines).  Just
> about any file format we might use would allow us to make "EAPI="
> appear in it, but not all could guarantee that it would occur at the
> start of a line, or at the start of a line immediately after a #.
>
> In any case, I can really see the KISS value in a very rigid syntax
> that is trivial to parse.  Stuff like this almost makes me wish our
> ebuilds already were xml files or such, with bash embedded inside
> sections.  Finding a particular tag in an xml file is trivial as the
> fundamentals haven't changed in 15 years.

I will stab the next person who suggests 'xml-like ebuilds.'

-A

>
> Rich
>

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Kent Fredric]

On 12 March 2012 15:24, Alec Warner <antarus@gentoo.org> wrote:

> I will stab the next person who suggests 'xml-like ebuilds.'

State-fully coded ebuilds, while perhaps not to your liking, for some
code-types can be incredibly useful.

For example, 9/10 perl-module ebuilds don't need any code at all in
the ebuild itself, it could do most of the work via structured data,
with the  rest of the work done by an e-class.

However, bash has serious problems when you want to represent hierarchical data.


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Kent Fredric]

On 12 March 2012 15:20, Rich Freeman <rich0@gentoo.org> wrote:
> On Sun, Mar 11, 2012 at 10:03 PM, Brian Harring <ferringb@gmail.com> wrote:
>> Pragmatic reality, the eapi function actually would work fine.  As
>> pointed out elsewhere, bash parses as it goes- which isn't going to
>> change.
>
> Unless the ebuild isn't written in bash...
>
> How do you source the ebuild if you don't know what to use to source
> it?  How do you know what to use to source it if you don't know the
> EAPI?  Right now all the existing EAPIs use bash, but there is no
> reason the file couldn't be xml, or python, or just about anything
> else.


A convention that is often used in this scenario is to combine a
hashbang call with that specific sourcer stripping that hashbang prior
to the parse.

ie:

foo.ebuild:

#!/usr/bin/env eapi-xml-5

would send the current file ( foo.ebuild ) to the process "eapi-xml-5"
( as defined by the current $PATH setting )

This process can easily then strip the #! stanza before sending the
content to an XML parser.

( ps. while I wouldn't actually use XML, its a good example case
because standard shell-script style commenting is illegal syntax in
XML )

The benefits of this approach seem obvious, but there are also obvious
downsides.

1. PRO: Unlike /usr/bin/eapi  , this style is PMS Agnostic, and only
requires the PMS have a process named "eapi-xml-5" *somewhere* in the
system under $PATH

2. CON: Unlike /usr/bin/eapi , you're limited by what you can send it
, /usr/bin/env  eapi xml-5 would be preferable syntax imo, but it
doesn't work, because its parsed as [ '/usr/bin/env' , 'eapi xml-5' ]
which then yeilds a "permission denied" due to no  command with that
name existing.

3. PRO: There's not /much/ risk of a user trying to run it directly,
mostly because there's no +x bit

4. CON: This syntax is going to conflict with whatever language we are
proceeding, which while this caveat is dealt with by the command that
the file is dispatched to, its a lot of work getting everything else
to play ball ( editors won't understand, linters wont understand,
various other tools that work strictly with the native forms of that
language wont understand ) and this problem exists for *every* case
where the coding system we're targeting doesn't natively support
whatever "magic" indicators we're trying to stuff in the file.

5. PRO: detecting this notation being used in a file is cheap-ish ,
you only have to read 2 characters into the file to know its using
this notation.


As a result of all these, it seems to me if we ever want to support
non-bash or non-scripting sources ( ie: YAML, JSON, XML , etc ) or any
other language which doesn't support "# is a comment" , we're going to
be pretty much forced to declare the EAPI outside the file somehow.

Whether this is via the filename ( yes, I've seen that debate ) or via
some other metadata file ( which seems more poison than cure ) would
be the real issue.

Having to store this in a metadata file that maps *.ebuild to EAPI
would seem like a solution that would only increase the filesize of
portage, and slow down runtime substantially, unless of course you had
a per-repository index that sped this up and was generated during the
metadata/ generation phase.

Then $PMS could just load that file from each repository , and
assuming the cache was still valid, it could easily know the EAPI for
any .ebuild it would source in advance.

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/11/2012 11:50 PM, Kent Fredric wrote:
> #!/usr/bin/env eapi-xml-5
> 
> would send the current file ( foo.ebuild ) to the process "eapi-xml-5"
> ( as defined by the current $PATH setting )

All we need is a way for the package manager to probe the EAPI. Spawning
a process to do that is just overkill.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Kent Fredric]

On 12 March 2012 20:08, Zac Medico <zmedico@gentoo.org> wrote:
> On 03/11/2012 11:50 PM, Kent Fredric wrote:
>> #!/usr/bin/env eapi-xml-5
>>
>> would send the current file ( foo.ebuild ) to the process "eapi-xml-5"
>> ( as defined by the current $PATH setting )
>
> All we need is a way for the package manager to probe the EAPI. Spawning
> a process to do that is just overkill.

Yeah, but if you read the rest of my message, you'll see my argument
that hinges around the fact most proposals so far tend to focus around
the fact the current format for ebuild is bash, and bash only, and  I
think it very hard to implement a language agnostic way to specify the
EAPI in a way that will work on languages where you

a) Can't use # to mark comments
b) May not even be working with the file directly, and perhaps
generating a .json file from a data-structure being serialised, and
the key named "EAPI" may be unpredictably placed near the end of the
.json file.

Case b is entirely plausible, and even realistic, because
non-script-based data-structures often don't bear importance on the
order of elements.


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

On Mon, 12 Mar 2012 00:08:46 -0700
Zac Medico <zmedico@gentoo.org> wrote:

> On 03/11/2012 11:50 PM, Kent Fredric wrote:
> > #!/usr/bin/env eapi-xml-5
> > 
> > would send the current file ( foo.ebuild ) to the process
> > "eapi-xml-5" ( as defined by the current $PATH setting )
> 
> All we need is a way for the package manager to probe the EAPI.
> Spawning a process to do that is just overkill.

And we could just use a good regex for that instead.

Something like: [eE][aA][pP][iI] <whitespace-or-symbols> [a-z0-9-+]+

and just require users for this to be the first thing declared in
an ebuild. Of course, this could make problems with stuff like:

# EAPI 4 because of foobarbaz
EAPI=4

(on the other hand, in this particular case it will fetch '4' anyway).

And this will work as well with:

<eapi>15-xml</eapi>

and

- eapi: 15-yaml

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

On Mon, 12 Mar 2012 09:27:11 +0100
Michał Górny <mgorny@gentoo.org> wrote:
> <eapi>15-xml</eapi>
> 
> and
> 
> - eapi: 15-yaml

You're carefully concocting your examples to make it look like it
should work. If you go the XML route, though, the EAPI would either be
in a DTD or as <eapi value="15" />.

Part of the point of all of this is that we shouldn't have to guess
what future EAPIs will look like.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 989 bytes --]

On Mon, 12 Mar 2012 08:30:19 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Mon, 12 Mar 2012 09:27:11 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> > <eapi>15-xml</eapi>
> > 
> > and
> > 
> > - eapi: 15-yaml
> 
> You're carefully concocting your examples to make it look like it
> should work.

Or I am just printing the first thing that comes into my head.

> If you go the XML route, though, the EAPI would either be in a DTD

Like .../gentoo/eapi/15-xml.dtd ? That would match my regex as well,
unless we allow dots in EAPI.

> or as <eapi value="15" />.

No, definitely not. That's not the XML style.

> Part of the point of all of this is that we shouldn't have to guess
> what future EAPIs will look like.

I'm just suggesting a way which will support a little more than
bash-based solutions. We could also assume that if a file doesn't match
the regexp at all, it's a unsupported EAPI.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Kent Fredric]

On 12 March 2012 22:09, Michał Górny <mgorny@gentoo.org> wrote:

>> or as <eapi value="15" />.
>
> No, definitely not. That's not the XML style.

Sure, but these examples are just examples after all. And XML is only
being used for an example use case, but there are many  more
structured formats than XML.

Some of us are mostly just worried that the proposals as they stand
won't be resilient enough to allow a future that isn't bash.

>> Part of the point of all of this is that we shouldn't have to guess
>> what future EAPIs will look like.
>
> I'm just suggesting a way which will support a little more than
> bash-based solutions. We could also assume that if a file doesn't match
> the regexp at all, it's a unsupported EAPI.

I just find a top-down regexp solution dangerously naive, as its
infering that the first line that matches the regexp *is* the EAPI
requirement field, when depending on the actual format used, that may
not be the case.

If for example, a format is machine generated, and the EAPI
declaration accidentally comes after something that *isnt* an EAPI
declaration but by the regexp, LOOKS like one,   then the probing
mechanism will resolve the WRONG value.

And that doesn't strike me as being very resilient.

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ulrich Mueller]

>>>>> On Mon, 12 Mar 2012, Kent Fredric wrote:

> I just find a top-down regexp solution dangerously naive, as its
> infering that the first line that matches the regexp *is* the EAPI
> requirement field, when depending on the actual format used, that
> may not be the case.

There's little danger if we require the EAPI specification to be in
the first line of the ebuild. Of course the regexp should be general
enough to account for a non-bash comment syntax.

Ulrich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Kent Fredric]

On 12 March 2012 22:48, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Mon, 12 Mar 2012, Kent Fredric wrote:
> There's little danger if we require the EAPI specification to be in
> the first line of the ebuild. Of course the regexp should be general
> enough to account for a non-bash comment syntax.
>

There's the obvious case of compiled-binaries where that might not be
possible, but thats definately strawman stuff and I wouldn't support
that sort of nonsense anyway. Compiled binaries for ebuilds can gtfo.

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Rich Freeman]

On Mon, Mar 12, 2012 at 6:12 AM, Kent Fredric <kentfredric@gmail.com> wrote:
>
> There's the obvious case of compiled-binaries where that might not be
> possible, but thats definately strawman stuff and I wouldn't support
> that sort of nonsense anyway. Compiled binaries for ebuilds can gtfo.
>

Why do I feel like a similar debate must have happened on some list
back when the SMTP message terminator was worked out?  The problem is
that we're running into is the use of in-band control information.

An advantage of sticking the EAPI in the filename is that it is
completely out-of-band.  Other options like an external file or POSIX
attributes have the same advantage (with the disadvantage that they
are harder to keep in-sync or support in general).

If you want to stick the EAPI inside the file then we have to go with
one of two routes:

1.  Make the file directly interpret-able by some existing tool like
bash, xml parsers, or (gtfo) ld-linux.so.  This requires making the
EAPI in-band info, and thus if the file format changes there is risk
that any existing code that interprets EAPIs could break, to the
degree that we weren't infinitely clever in designing the spec.

2.  Make the file require pre-processing before being fed to another
interpreter.  This breaks syntax-highlighting and such, and is
generally more of a pita.  However, this allows in-file EAPI info to
remain out-of-band.  An ebuild might be an xml file with EAPI and
other meta-data, and might contain sections that get ripped out and
handed over to bash, python, or ld-linux.so to the extent that you
want to be told to gtfo.

In my thinking the option with the fewest drawbacks and the greatest
extent of future-proofing is to go with GLEP 55 and stick it in the
filename.  Every filesystem and rsync tool handles filenames, they're
impossible to separate from file contents, and every interpreter out
there ignores them.

If we don't want to stick it in the filename then either we have to
decide that whatever we come up with could break, or that we really
have to change what ebuilds look like.

Rich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/12/2012 02:16 AM, Kent Fredric wrote:
> I just find a top-down regexp solution dangerously naive, as its
> infering that the first line that matches the regexp *is* the EAPI
> requirement field, when depending on the actual format used, that may
> not be the case.
> 
> If for example, a format is machine generated, and the EAPI
> declaration accidentally comes after something that *isnt* an EAPI
> declaration but by the regexp, LOOKS like one,   then the probing
> mechanism will resolve the WRONG value.
> 
> And that doesn't strike me as being very resilient.

It would be very fragile without the sanity check / feedback mechanism
that's already been suggested [1]. The idea is to compare the probed
EAPI with the result that's obtained from bash, and treat the ebuild as
invalid if they are not identical. This would allow ebuilds that don't
fit our EAPI probing mechanism to be immediately detected so that the
ebuild developer could fix them. In practice, this will not be much of
an issue, since the EAPI assignment will be required to be very close to
the top of the file, and the package manager will stop searching as soon
as it finds the first match.

[1] https://bugs.gentoo.org/show_bug.cgi?id=402167#c18
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Rich Freeman]

On Mon, Mar 12, 2012 at 1:01 PM, Zac Medico <zmedico@gentoo.org> wrote:
> It would be very fragile without the sanity check / feedback mechanism
> that's already been suggested.

Another obvious check is to have repoman run a grep with the regexp
and give an error if there is not exactly one match.

Rich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/12/2012 10:30 AM, Rich Freeman wrote:
> On Mon, Mar 12, 2012 at 1:01 PM, Zac Medico <zmedico@gentoo.org> wrote:
>> It would be very fragile without the sanity check / feedback mechanism
>> that's already been suggested.
> 
> Another obvious check is to have repoman run a grep with the regexp
> and give an error if there is not exactly one match.

If we want to handle every possible screwup, including stray EAPI
assignments inside inherited eclasses, we still need to compare the
probed value to the value that's obtained from bash.

I guess you're hinting at using a non-bash ebuild format? Even if in
that case, the the package manager should simply use whatever
interpreter is appropriate for the probed EAPI. So, if EAPI 7 is a
posix-shell format, and the probed EAPI is 7, then the package manager
should source the ebuild with it's posix-shell instead of bash. If the
probed EAPI is not a supported EAPI, then it should skip the sourcing
entirely, and report the ebuild as having an unsupported EAPI.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Rich Freeman]

On Mon, Mar 12, 2012 at 1:46 PM, Zac Medico <zmedico@gentoo.org> wrote:
> If we want to handle every possible screwup, including stray EAPI
> assignments inside inherited eclasses, we still need to compare the
> probed value to the value that's obtained from bash.

Well, I wasn't intending to suggest that the repoman check need be the
only one.  However, preventing problems is at least as useful as
detecting them.

That said, it would probably best to have exactly one way to determine
the official EAPI.  If that is to parse the filename, then parse the
filename.  If it is to grep for a regexp and expect exactly one hit
and parse it in a certain way, then do that.

It might be that the "one" official way is to grep for a regexp and if
you find it, use it, otherwise assume 0, 1, 2, 3, or 4 and source the
ebuild for it.  That still gives you only a single answer (well,
except in situations where the current way is already broken).

If people want to abuse the EAPI syntax I suppose we can generate an
error, but ignoring it might be just as valid behavior.  I'm not sure
what happens if you define PN/etc in ebuild besides things breaking in
a horrible manner.  I'd put changing EAPI in the same category.

Rich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Kent Fredric]

On 12 March 2012 21:27, Michał Górny <mgorny@gentoo.org> wrote:
> And we could just use a good regex for that instead.
>
> Something like: [eE][aA][pP][iI] <whitespace-or-symbols> [a-z0-9-+]+
>
> and just require users for this to be the first thing declared in
> an ebuild. Of course, this could make problems with stuff like:
>
> # EAPI 4 because of foobarbaz
> EAPI=4
>
> (on the other hand, in this particular case it will fetch '4' anyway).
>
> And this will work as well with:
>
> <eapi>15-xml</eapi>
>
> and
>
> - eapi: 15-yaml
>

Also, remember the proposal is to read it only from the first 10-30
lines of the file, and if you're *generating* YAML/XML , then this is
not necessarily guaranteed.

Some generation tools emit keys in alphanumeric ordering, others
psuedo-randomly, and the EAPI declaration line in some formats could
easily be on the very last line of the file.

And then the regexp could falsely detect something in another key the
original author had not intended as an EAPI definition, but merely a
comment, ...

ie:

<generateddata>
    <comment>  This is a user comment, hurr, I hate EAPI 5 </comment>
    <eapi value="15" />
</generateddata>


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1296 bytes --]

On Mon, 12 Mar 2012 21:39:52 +1300
Kent Fredric <kentfredric@gmail.com> wrote:

> On 12 March 2012 21:27, Michał Górny <mgorny@gentoo.org> wrote:
> > And we could just use a good regex for that instead.
> >
> > Something like: [eE][aA][pP][iI] <whitespace-or-symbols> [a-z0-9-+]+
> >
> > and just require users for this to be the first thing declared in
> > an ebuild. Of course, this could make problems with stuff like:
> >
> > # EAPI 4 because of foobarbaz
> > EAPI=4
> >
> > (on the other hand, in this particular case it will fetch '4'
> > anyway).
> >
> > And this will work as well with:
> >
> > <eapi>15-xml</eapi>
> >
> > and
> >
> > - eapi: 15-yaml
> >
> 
> Also, remember the proposal is to read it only from the first 10-30
> lines of the file, and if you're *generating* YAML/XML , then this is
> not necessarily guaranteed.
> 
> Some generation tools emit keys in alphanumeric ordering, others
> psuedo-randomly, and the EAPI declaration line in some formats could
> easily be on the very last line of the file.

You should ensure your tool does the right thing, I'd say. XML with DTD
often requires specific element order, so such a tool is simply bound
to fail with XML.

For yaml, it's a fair point.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/11/2012 07:03 PM, Brian Harring wrote:
> Pragmatic reality, the eapi function actually would work fine.  As 
> pointed out elsewhere, bash parses as it goes- which isn't going to 
> change.
> 
> If someone invokes 'eapi happy-meal' and it's not supported, 
> the sourcing is stopped immediately, cache gets -happy-meal for the 
> EAPI, and the pm continues to ignore the ebuild till either the 
> ebuilds mtime changes (which case it redoes the metadata regen) or the 
> PM now supports that EAPI, and... you guessed it, redoes the metadata 
> regen.  The cache behaviour is basically the same regardless of the 
> EAPI mechanism.
> 
> Now, this isn't to say everyone views the function as *optimal*.  
> People can argue about that as much as they'd like.
> 
> The point however is that it *would* work.  Anyone claiming 
> fragility/otherwise needs to put forth actual code examples.

Suppose that EAPI 5 requires bash-5. There may be bash-5 syntax in the
ebuild prior to your eapi() function call, causing a fatal syntax error.

>> Anyway, lets focus on our main goal, which is to decide on a way to
>> obtain the EAPI _without_ sourcing the ebuild.
> 
> Nitpicking, but the point needs be made; this is *your* requirement.  
> The requirement is to be able to deploy new globals/bash 
> requirements/whatever.

Well, I think most people would tend to accept my requirement, and that
it falls within the realm of common-sense if avoiding unnecessary
complexity is one of our goals.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1332 bytes --]

On Fri, 09 Mar 2012 12:31:24 -0500
Michael Orlitzky <michael@orlitzky.com> wrote:

> On 03/09/12 12:11, Ulrich Mueller wrote:
> >>>>>> On Fri, 09 Mar 2012, Michael Orlitzky wrote:
> > 
> >>> What if bash starts to parse the script completely and barfs at
> >>> 'syntax error' before it starts executing stuff?
> > 
> >> It doesn't parse the script completely, it executes line-by-line,
> >> so we can bail out early.
> > 
> > How can you tell that this behaviour won't be changed in a future
> > bash version?
> > 
> 
> Who's to say that in the future my computer won't be made out of
> delicious ice cream, eliminating the need for EAPIs entirely?
> 
> Chances are, this would break thousands of scripts, so we hope they
> wouldn't do it. If it does happen, we either deal with it then, or
> don't upgrade to that version of bash -- the same as we would do with
> any other massive breaking change.

Thousands of scripts? So... you're saying that people actually use
thousands of scripts which have invalid syntax...

Well, one thing I can think of now is makeself and similar. Those are
indeed a quite good argument.

But the main point here is that at some point someone may want to use
a non-bash syntax for ebuilds. Or some kind of optional bash extension.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Brian Harring]

On Fri, Mar 09, 2012 at 06:52:40PM +0100, Micha?? G??rny wrote:
> On Fri, 09 Mar 2012 12:31:24 -0500
> Michael Orlitzky <michael@orlitzky.com> wrote:
> 
> > On 03/09/12 12:11, Ulrich Mueller wrote:
> > >>>>>> On Fri, 09 Mar 2012, Michael Orlitzky wrote:
> > > 
> > >>> What if bash starts to parse the script completely and barfs at
> > >>> 'syntax error' before it starts executing stuff?
> > > 
> > >> It doesn't parse the script completely, it executes line-by-line,
> > >> so we can bail out early.
> > > 
> > > How can you tell that this behaviour won't be changed in a future
> > > bash version?
> > > 
> > 
> > Who's to say that in the future my computer won't be made out of
> > delicious ice cream, eliminating the need for EAPIs entirely?
> > 
> > Chances are, this would break thousands of scripts, so we hope they
> > wouldn't do it. If it does happen, we either deal with it then, or
> > don't upgrade to that version of bash -- the same as we would do with
> > any other massive breaking change.
> 
> Thousands of scripts? So... you're saying that people actually use
> thousands of scripts which have invalid syntax...

Just a note; you need to look into how aliases work.  That right there 
unfortunately means bash isn't going to pre-parse, not as long as 
aliases are supported.

Back to your arguing...
~brian

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[James Broadhead]

On 9 March 2012 17:31, Michael Orlitzky <michael@orlitzky.com> wrote:
> At any rate, I'm now convinced that we all want GLEP 55, but with a
> different name.

I think that moving the data to the filename is probably a better
approach than semi- or repeat parsing, but I prefer preserving the
.ebuild extension, and think that eapi should be specified similarly
to ebuild revision, as a suffix. for instance:

app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the
new schema)
app-foo/bar-1.0.0-r1-e1.ebuild
app-foo/bar-1.0.0-r1-e99.ebuild

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alexis Ballier]

On Fri, 9 Mar 2012 18:02:51 +0000
James Broadhead <jamesbroadhead@gmail.com> wrote:

> On 9 March 2012 17:31, Michael Orlitzky <michael@orlitzky.com> wrote:
> > At any rate, I'm now convinced that we all want GLEP 55, but with a
> > different name.
> 
> I think that moving the data to the filename is probably a better
> approach than semi- or repeat parsing, but I prefer preserving the
> .ebuild extension, and think that eapi should be specified similarly
> to ebuild revision, as a suffix. for instance:
> 
> app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the
> new schema)
> app-foo/bar-1.0.0-r1-e1.ebuild
> app-foo/bar-1.0.0-r1-e99.ebuild
> 

if you want to keep .ebuild you need to keep current naming, afaik
package managers fail on invalid names

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 10:24 AM, Alexis Ballier wrote:
> On Fri, 9 Mar 2012 18:02:51 +0000
> James Broadhead <jamesbroadhead@gmail.com> wrote:
> 
>> On 9 March 2012 17:31, Michael Orlitzky <michael@orlitzky.com> wrote:
>>> At any rate, I'm now convinced that we all want GLEP 55, but with a
>>> different name.
>>
>> I think that moving the data to the filename is probably a better
>> approach than semi- or repeat parsing, but I prefer preserving the
>> .ebuild extension, and think that eapi should be specified similarly
>> to ebuild revision, as a suffix. for instance:
>>
>> app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the
>> new schema)
>> app-foo/bar-1.0.0-r1-e1.ebuild
>> app-foo/bar-1.0.0-r1-e99.ebuild
>>
> 
> if you want to keep .ebuild you need to keep current naming, afaik
> package managers fail on invalid names

Invalid names like those should only trigger warnings.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/09/12 13:02, James Broadhead wrote:
> On 9 March 2012 17:31, Michael Orlitzky <michael@orlitzky.com> wrote:
>> At any rate, I'm now convinced that we all want GLEP 55, but with a
>> different name.
> 
> I think that moving the data to the filename is probably a better
> approach than semi- or repeat parsing, but I prefer preserving the
> .ebuild extension, and think that eapi should be specified similarly
> to ebuild revision, as a suffix. for instance:
> 
> app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the
> new schema)
> app-foo/bar-1.0.0-r1-e1.ebuild
> app-foo/bar-1.0.0-r1-e99.ebuild
> 

One of the benefits of GLEP 55 naming is that old package managers won't
try to parse them. So, for example, if we put new features in,

  app-foo/bar-1.0.0.ebuild-5

portage from 2003 won't try to source it.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/09/2012 10:33 AM, Michael Orlitzky wrote:
> On 03/09/12 13:02, James Broadhead wrote:
>> On 9 March 2012 17:31, Michael Orlitzky <michael@orlitzky.com> wrote:
>>> At any rate, I'm now convinced that we all want GLEP 55, but with a
>>> different name.
>>
>> I think that moving the data to the filename is probably a better
>> approach than semi- or repeat parsing, but I prefer preserving the
>> .ebuild extension, and think that eapi should be specified similarly
>> to ebuild revision, as a suffix. for instance:
>>
>> app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the
>> new schema)
>> app-foo/bar-1.0.0-r1-e1.ebuild
>> app-foo/bar-1.0.0-r1-e99.ebuild
>>
> 
> One of the benefits of GLEP 55 naming is that old package managers won't
> try to parse them. So, for example, if we put new features in,
> 
>   app-foo/bar-1.0.0.ebuild-5
> 
> portage from 2003 won't try to source it.

Every software product has an end of life. I think if a system hasn't
been updated in the last 2 years or so, then it's fair to assume that it
will never be updated. So, all relevant versions of portage should
simply show a warning message if the encounter an ebuild name such as
"app-foo/bar-1.0.0-r1-e99.ebuild".
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

On Fri, 09 Mar 2012 10:56:03 -0800
Zac Medico <zmedico@gentoo.org> wrote:
> Every software product has an end of life. I think if a system hasn't
> been updated in the last 2 years or so, then it's fair to assume that
> it will never be updated. So, all relevant versions of portage should
> simply show a warning message if the encounter an ebuild name such as
> "app-foo/bar-1.0.0-r1-e99.ebuild".

It's worth noting that "showing a warning message" will result in
zillions of freaking out users, and that we want to be able to
introduce new EAPIs without having a stable Portage version that
handles it around.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/09/12 13:56, Zac Medico wrote:
> On 03/09/2012 10:33 AM, Michael Orlitzky wrote:
>> On 03/09/12 13:02, James Broadhead wrote:
>>> On 9 March 2012 17:31, Michael Orlitzky <michael@orlitzky.com> wrote:
>>>> At any rate, I'm now convinced that we all want GLEP 55, but with a
>>>> different name.
>>>
>>> I think that moving the data to the filename is probably a better
>>> approach than semi- or repeat parsing, but I prefer preserving the
>>> .ebuild extension, and think that eapi should be specified similarly
>>> to ebuild revision, as a suffix. for instance:
>>>
>>> app-foo/bar-1.0.0-r1.ebuild # EAPI0 (or the highest EAPI prior to the
>>> new schema)
>>> app-foo/bar-1.0.0-r1-e1.ebuild
>>> app-foo/bar-1.0.0-r1-e99.ebuild
>>>
>>
>> One of the benefits of GLEP 55 naming is that old package managers won't
>> try to parse them. So, for example, if we put new features in,
>>
>>   app-foo/bar-1.0.0.ebuild-5
>>
>> portage from 2003 won't try to source it.
> 
> Every software product has an end of life. I think if a system hasn't
> been updated in the last 2 years or so, then it's fair to assume that it
> will never be updated. So, all relevant versions of portage should
> simply show a warning message if the encounter an ebuild name such as
> "app-foo/bar-1.0.0-r1-e99.ebuild".

I was just repeating your point against the eapi function =)

And there is a non-zero benefit to it, I've had to rescue systems that
haven't seen an update in years.

The best reason I can think of for using ebuild-EAPI is simply semantic.
The basename of the ebuild refers to the package, the extension decides
what interprets it. That is at least consistent with every other file
extension.

[gentoo-dev] Re: RFD: EAPI specification in ebuilds

[Steven J Long]

Firstly, wrt probing the ebuild for EAPI=.. I'd just like to point out that 
a regex is not required during the scan, and nor is restricting it to the 
first N lines, though the latter may be desirable and could trivially 
exclude comment and whitespace-only or empty lines.

Ciaran McCreesh wrote:
>Michael Orlitzky wrote:
>> Fair enough, but aren't you arguing the opposite point with Zac? If
>> ebuilds are data, fine, we write EAPI=4 somewhere and be done with
>> it. Anything not having that format is out-of-spec.
> 
> The problem is that right now there's no way to determine the format of
> the data until you already know the format of the data.
Well, we know it's bash.. ;)

> We hack around
> this by not allowing "drastic" format changes, where "drastic" includes
> "using things in newer versions of bash" and "not adding new global
> scope commands".
> 
> The question under discussion is whether we a) keep "what format the
> data is in" as being part of the data, but impose some strange and
> arbitrary conditions on it
Stipulating an allowed set of characters is in no way arbitrary, nor 
strange- we already have similar restrictions on category and package names, 
versions, keywords and USE flags, for example. Requiring that the EAPI 
assignment for a bash .ebuild must be a literal (ie EAPI="foo" or EAPI=foo 
or EAPI='foo') at the start of a line, is not hard to understand; as you 
said ebuild authors already have to deal with lots of other subtle 
restrictions. As Marc Schiffbauer said, EAPI "might be the most important 
constraint in an ebuild at all" (from this and earlier discussion, it's 
clear that it definitely is) -- ebuild developers have to know about it, and 
this is a simple, clear restriction. Michał Górny stated: "The most 
important ebuild variables like EAPI should be readable on sight, without 
having to lookup random variables, functions etc" and in fact, all ebuilds 
in the tree already use a string literal- it just makes more sense from a 
code readability pov, quite apart from anything else.

You mentioned indentation in another mail; afaics there is no problem with 
whitespace at the start of the line.

Ensuring EAPI declaration and sourced value match, has value beyond use in 
EAPI extraction: it's a sanity check that should be performed in any case, 
way before an ebuild hits the tree.

> , b) make a one-time change to have some kind
> of 'header' inside the file describing its format that isn't really part
> of the data itself, or c) admit that GLEP 55 already solved the problem
> and we might as well just fix the issue properly once and for all, even
> if GLEP 55's author is considered by some to be one of Satan's little
> minions.
> 
Regardless of your past behaviour, my objections to GLEP 55 are, and always 
have been, technical: it breaks encapsulation, which once implemented cannot 
be taken back. It results in a mess of filename extensions, which are 
confusing and irrelevant to end-users, as well as making other tools and 
scripts trickier to implement; a simple example: a highlighting editor would 
need to pattern-match the file extension. It's not needed: the simplest, 
least-invasive alternative already works, and should have been adopted years 
ago, when the Council asked for alternatives to be tried. The tree is 
clearly in shape to do so now, though.

Package versions have to be in the filename to avoid collisions, and indeed 
the information is relevant to both end-users and developers. EAPI, while 
vital to the mangler and of immediate concern to developers, matches neither 
of those. Since it is of immediate concern, restricting it to a string 
literal makes sense from both maintenance (which is why it matches tree-
usage) and implementation perspectives. And specifying what characters are 
allowed is a no-brainer; it's odd that that still has not been done, despite 
it also being a requirement for embedding EAPI in filenames.

Your motivation for GLEP-55 states: "In order to get the EAPI the package 
manager needs to source the ebuild." Given a suitable specification, that 
isn't the case. repoman checks and explicit documentation are all that's 
needed beyond that.

As for non-bash ebuilds, I have always agreed with antarus that they should 
simply use a different extension. Adding a new extension per source language 
is a *lot* cleaner than one per EAPI.

Regards,
Steve.
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: RFD: EAPI specification in ebuilds

[Kent Fredric]

On 19 March 2012 14:12, Steven J Long <slong@rathaus.eclipse.co.uk> wrote:
>
> As for non-bash ebuilds, I have always agreed with antarus that they should
> simply use a different extension. Adding a new extension per source language
> is a *lot* cleaner than one per EAPI.

Ok: If we take this notion and enshrine it in stone:

If we assume Bash 4 is a seperate language from Bash 3, as its
syntax-backwards-incompatible, is it fair to suggest that for some
future EAPI which require Bash 4, that the extension change to suit?

ie:  move from .ebuild  to .ebuild4 , where '.ebuild' conveys the
format is bash, and that '.ebuild4' is bash4 only?

That way you have a forwards declaration of the syntax/file format
required to parse the file, but no declaration of the EAPI, so you're
not breaking encapsulation.

This is breaking the direct file==eapi connection, but still
maintaining a loose file<->eapi connection.

Its /sort/ of like the "one time extension change" proposal, except
its less 'arbitrary' than something like .eb , and it gives us the
future option of changing the suffix again if bash 5 comes out with
different syntax.

Then we can do

.ebuild = EAPI 0 - 4    & bash >= 3
.ebuild4 = EAPI5 - 9    & bash >= 4
.ebuild5 = EAPI10 - 15 & bash >= 5

Thoughts?

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] Re: RFD: EAPI specification in ebuilds

[Brian Harring]

On Mon, Mar 19, 2012 at 02:36:34PM +1300, Kent Fredric wrote:
> On 19 March 2012 14:12, Steven J Long <slong@rathaus.eclipse.co.uk> wrote:
> >
> > As for non-bash ebuilds, I have always agreed with antarus that they should
> > simply use a different extension. Adding a new extension per source language
> > is a *lot* cleaner than one per EAPI.
> 
> Ok: If we take this notion and enshrine it in stone:
> 
> If we assume Bash 4 is a seperate language from Bash 3, as its
> syntax-backwards-incompatible, is it fair to suggest that for some
> future EAPI which require Bash 4, that the extension change to suit?
> 
> ie:  move from .ebuild  to .ebuild4 , where '.ebuild' conveys the
> format is bash, and that '.ebuild4' is bash4 only?
> 
> That way you have a forwards declaration of the syntax/file format
> required to parse the file, but no declaration of the EAPI, so you're
> not breaking encapsulation.
> 
> This is breaking the direct file==eapi connection, but still
> maintaining a loose file<->eapi connection.
> 
> Its /sort/ of like the "one time extension change" proposal, except
> its less 'arbitrary' than something like .eb , and it gives us the
> future option of changing the suffix again if bash 5 comes out with
> different syntax.
> 
> Then we can do
> 
> .ebuild = EAPI 0 - 4    & bash >= 3
> .ebuild4 = EAPI5 - 9    & bash >= 4
> .ebuild5 = EAPI10 - 15 & bash >= 5
> 
> Thoughts?

This is a bad idea; it arbitrarily bleeds the bash version into the 
ebuild name, still requires an EAPI mechanism w/in the actual file, 
and is likely to break tools that have assumptions about extensions 
(even ones sooner or later written against such a setup).

Besides; it's not bash version as much as global scope settings, 
functions, etc, that are the issue.

Syntax is generally minor in comparison.
~harring

[gentoo-dev] Re: Re: RFD: EAPI specification in ebuilds

[Steven J Long]

Kent Fredric wrote:
> On 19 March 2012 14:12, Steven J Long <slong@rathaus.eclipse.co.uk> wrote:
>>
>> As for non-bash ebuilds, I have always agreed with antarus that they
>> should simply use a different extension. Adding a new extension per
>> source language is a *lot* cleaner than one per EAPI.
> 
> Ok: If we take this notion and enshrine it in stone:
> 
> If we assume Bash 4 is a seperate language from Bash 3, as its
> syntax-backwards-incompatible, is it fair to suggest that for some
> future EAPI which require Bash 4, that the extension change to suit?
>
I don't agree that it's a separate language: a lot of work goes into keeping 
BASH backward-compatible, so it's only when you use a new feature that your 
script will no longer work on an older version. We dealt with the move from 
bash2 to bash 3 quite well, I thought, and I don't see what's wrong with 
stipulating support for a new BASH version in a new EAPI: that seems 
quintessential use of the EAPI mechanism to me.

Another example would be use of extglob: as a scripting feature it needs to 
be enabled before any of the functions or script using it are even parsed, 
or BASH will bail with a syntax error. Specifying that it is so enabled in 
an EAPI (which I believe was done a while back) allows the ebuild author to 
use syntax that would otherwise fall over. Again, the EAPI mechanism works 
fine, and there is no need for a different extension, even though the syntax 
accepted has changed.

So we already have an example of where syntax changes, not just the 
available environment, work with an EAPI inside the file, as it should be.

> ie:  move from .ebuild  to .ebuild4 , where '.ebuild' conveys the
> format is bash, and that '.ebuild4' is bash4 only?
> 
> That way you have a forwards declaration of the syntax/file format
> required to parse the file, but no declaration of the EAPI, so you're
> not breaking encapsulation.
> 
> This is breaking the direct file==eapi connection, but still
> maintaining a loose file<->eapi connection.
> 
> Its /sort/ of like the "one time extension change" proposal, except
> its less 'arbitrary' than something like .eb , and it gives us the
> future option of changing the suffix again if bash 5 comes out with
> different syntax.
> 
> Then we can do
> 
> .ebuild = EAPI 0 - 4    & bash >= 3
> .ebuild4 = EAPI5 - 9    & bash >= 4
> .ebuild5 = EAPI10 - 15 & bash >= 5
> 
> Thoughts?
>
As Brian said, you still need an EAPI mechanism, and further you've still 
leaked metadata that simply doesn't need to be leaked. The only thing that 
has ever held up adoption of EAPI extraction using a simple EAPI="foo" 
declaration, matching tree-usage, is the fact that PMS does not specify 
which characters are allowed in an EAPI name. This was asked for a couple of 
years ago, iirc, and it is clearly necessary whether you scan for a token in 
the ebuild, or you stick EAPI into the filename. (After all you are even 
more restricted when you're talking about the filename, as opposed to a text 
string given in a script.)

It just seems like a major hole in the spec, given that EAPI is so critical 
to how you deal with the ebuild, and characters allowed for other tokens 
such as category, package, version, keywords, USE-flags etc are specified. 
It would be necessary for GLEP-55, but is not mentioned therein, which I 
guess means the GLEP is under-specified as well.

.eapi-NN just strikes me as similar to having M$ Word format as .doc95 
.doc2000 and the like- a complete mess. Or indeed sticking the charset 
encoding (which has profound consequences when reading a file) from an HTML 
file into its filename. Again, a similar, if not worse, piece of metadata-- 
just extracting it is hard if you don't know the charset in advance, but 
I've never heard of anyone arguing that it should belong anywhere other than 
in the file, from where tools dealing with it locally can extract it and, 
say, emit the correct HTTP header.

What I was referring to with an extension-per-language would be something 
like .pbuild for python-based build scripts, or .xbuild for your XML-
declarative example, and so on. In each case, you'd still need a supporting 
infrastructure, but it's clear that you are using a completely different 
setup, and it's no longer a traditional BASH-based ebuild. It would be quite 
reasonable to suppose that you'd have EAPI specified there as well, since 
you'd want to be able to add to and change the features you support over 
time. Would you really want to change the extension each time you did, 
though?

Regards,
Steve.
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alexandre Rostovtsev]

On Wed, 2012-03-07 at 21:41 +0100, Ulrich Mueller wrote:
>     .ebuild -> .eb

FYI, any Russian speaker is *guaranteed* to read the name ".eb" as a
very common obscenity.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Jeroen Roovers]

On Wed, 07 Mar 2012 17:36:05 -0500
Alexandre Rostovtsev <tetromino@gentoo.org> wrote:

> FYI, any Russian speaker is *guaranteed* to read the name ".eb" as a
> very common obscenity.

In Dutch it means the low tide, and as a verb, it means "becoming low"
or "decreasing" as in the tide or some other fluid.

In English it means something very similar as in the Dutch verb, only
without the tidal reference. Other Germanic languages will probably
have their own variants.

In Crimean Tatar it apparently[1] references "easiness" or "simplicity",
in Hungarian it's the word for "dog", while in Nauruan it means
"country."

But that's not important right now (see also [2]).

The important thing to remember is, the bike shed protects our bicycles
from the elements.


     jer


[1] http://en.wiktionary.org/wiki/eb
[2] http://en.wikipedia.org/wiki/bytesex

[gentoo-dev] Ebb (eb) was: Re: RFD: EAPI specification in ebuilds

[Todd Goodman]

* Jeroen Roovers <jer@gentoo.org> [120307 21:25]:
> On Wed, 07 Mar 2012 17:36:05 -0500
> Alexandre Rostovtsev <tetromino@gentoo.org> wrote:
> 
> > FYI, any Russian speaker is *guaranteed* to read the name ".eb" as a
> > very common obscenity.
> 
> In Dutch it means the low tide, and as a verb, it means "becoming low"
> or "decreasing" as in the tide or some other fluid.
> 
> In English it means something very similar as in the Dutch verb, only
> without the tidal reference. Other Germanic languages will probably
> have their own variants.

It has the same meaning in English as well.  The tide can ebb.
Also, there are ebb tides (and neap tides as well.)

Todd

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alec Warner]

On Wed, Mar 7, 2012 at 12:41 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
> Hi all,
>
> The way how we currently specify the EAPI in ebuilds has some
> problems. For example, there is no sane way to allow usage of features
> of a new bash version in a new EAPI. So we are currently stuck with
> bash 3.2. Also changes of global scope behaviour, like addition of new
> global scope functions (similar to "inherit") are not possible.
>
> These flaws are outlined in GLEP 55 [1]:
> | In order to get the EAPI the package manager needs to source the
> | ebuild, which itself needs the EAPI in the first place. Otherwise it
> | imposes a serious limitation, namely every ebuild, using any of the
> | future EAPIs, will have to be source'able by old package managers
> | [...]
>
> The council has voted down GLEP 55 more than a year ago, but at the
> same time requested that a solution for the mentioned issues should be
> found. [2] However, there was no progress since then.
>
> The issue arose again in bug 402167 [3] where several solutions have
> been discussed. Below, I try to summarise the possible options
> resulting from that discussion.
>
>
> *** Proposal 1: "Parse the EAPI assignment statement" ***
>
> This first proposal would require that the syntax of the EAPI
> assignment statement in ebuilds matches a well defined regular
> expression. A scan of the Portage tree shows that the statement only
> occurs in the following variations (using EAPI 4 as example):
>
>   EAPI=4
>   EAPI="4"
>   EAPI='4'
>
> Sometimes this is followed by whitespace or a comment (starting with
> a # sign). Also, with very few exceptions the EAPI assignment occurs
> within the first few lines of the ebuild. For the vast majority of
> ebuilds it is in line 5.
>
> Written in a more formal way, appropriate for a specification:
> - Ebuilds must contain at most one EAPI assignment statement.
> - It must occur within the first N lines of the ebuild (N=10 and N=30
>  have been suggested).
> - The statement must match the following regular expression (extended
>  regexp syntax):
>  ^[ \t]*EAPI=(['"]?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$
>
> Note: The first and the third point are already fulfilled by all
> ebuilds in the Portage tree. The second point will require very few
> ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).
>
> The package manager would determine the EAPI by parsing the assignment
> with above regular expression. A sanity check would be added. Citing
> Zac Medico in [3]: "The fact that we can compare the probed EAPI to
> the actual EAPI variable after the ebuild is sourced seems like a
> perfect sanity check. We could easily detect inconsistencies and flag
> such ebuilds as invalid, providing a reliable feedback mechanism to
> ebuild developers."
>
> This proposal comes in two variants:
> 1a) The change is applied retroactively for all EAPIs.
> 1b) It is only applied for EAPI 5 and later (which means that the
>    result of the EAPI parsing would be discarded for earlier EAPIs).

I don't like this idea because the sane way should be easy and straightforward.
Mixing a constant declaration with bash assignment just confuses users
who think the assignment is full bash when in fact it is not.

EAPI=$(somefunc)
EAPI=${SOMEVAR%%-*}
and so forth all don't meet the regex (and would be flagged invalid.)
However a naive author might think they work.

I don't think any naive author would think either would work in a comment

# EAPI=$(somefunc)
# EAPI=${SOEMVAR%%-*}

Bash doesn't parse comments, so confusion is unlikely.

>
>
> *** Proposal 2: "EAPI in header comment" ***
>
> A different approach would be to specify the EAPI in a specially
> formatted comment in the ebuild's header. No syntax has been suggested
> yet, but I believe that the following would work as a specification:
> - The EAPI must be declared in a special comment in the first line of
>  the ebuild's header, as follows:
> - The first line of the ebuild must contain the word "ebuild",
>  followed by whitespace, followed by the EAPI, followed by
>  end-of-line or whitespace.
>
> Again, the proposal comes in two variants:
> 2a) It is combined with a one time change of the file extension, like
>    .ebuild -> .eb.
> 2b) The usual EAPI assignment statement in the ebuild is still
>    required, at least for a transition period.
>
> In the 2a case, the EAPI variable could be made read-only in bash
> before sourcing the ebuild. In the 2b case, a sanity check similar to
> the one mentioned above would be added.
>
>
> What do you think?

Overloading is bad.

There is no real difference between:
#!/usr/bin/ebuild --eapi 5
# EAPI=5

The problem is that the former is also the way to specify how to
execute the ebuild; so unless you plan to make ebuilds executable and
having /usr/bin/ebuild provide the ebuild environment, using that
syntax is confusing to users.

>
> (I really hope for a constructive discussion here. So, if you want
> to comment that all of the above proposals suck and GLEP 55 is much
> superior, then please open a new thread for it.)

You don't mention voting on glep 55 again; is there a reason why not?

>
> Ulrich
>
>
> [1] <http://www.gentoo.org/proj/en/glep/glep-0055.html#problem>
> [2] <http://www.gentoo.org/proj/en/council/meeting-logs/20100823-summary.txt>
> [3] <https://bugs.gentoo.org/show_bug.cgi?id=402167>
>

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ulrich Mueller]

>>>>> On Wed, 7 Mar 2012, Alec Warner wrote:

>> *** Proposal 1: "Parse the EAPI assignment statement" ***
>> [...]

> I don't like this idea because the sane way should be easy and
> straightforward. Mixing a constant declaration with bash assignment
> just confuses users who think the assignment is full bash when in
> fact it is not.

> EAPI=$(somefunc)
> EAPI=${SOMEVAR%%-*}
> and so forth all don't meet the regex (and would be flagged
> invalid.) However a naive author might think they work.

Such constructs also cannot be used with any of the other proposed
solutions. And in fact, nobody is using such things in practice.
_All_ ebuilds in the Portage tree can be successfully parsed with the
regexp proposed.

The obvious sanity check, i.e. comparing the EAPI obtained by parsing
and by sourcing, could be added to repoman, which would prevent such
non-conforming ebuilds from being committed to the tree.

>> *** Proposal 2: "EAPI in header comment" ***
>> [...]

> Overloading is bad.

> There is no real difference between:
> #!/usr/bin/ebuild --eapi 5
> # EAPI=5

> The problem is that the former is also the way to specify how to
> execute the ebuild; so unless you plan to make ebuilds executable and
> having /usr/bin/ebuild provide the ebuild environment, using that
> syntax is confusing to users.

I agree with this point.

Many file formats are identifying themselves with a magic token (as
it is used by sys-apps/file), but it is not necessarily a shebang.

Ulrich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alec Warner]

On Wed, Mar 7, 2012 at 11:27 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Wed, 7 Mar 2012, Alec Warner wrote:
>
>>> *** Proposal 1: "Parse the EAPI assignment statement" ***
>>> [...]
>
>> I don't like this idea because the sane way should be easy and
>> straightforward. Mixing a constant declaration with bash assignment
>> just confuses users who think the assignment is full bash when in
>> fact it is not.
>
>> EAPI=$(somefunc)
>> EAPI=${SOMEVAR%%-*}
>> and so forth all don't meet the regex (and would be flagged
>> invalid.) However a naive author might think they work.
>
> Such constructs also cannot be used with any of the other proposed
> solutions. And in fact, nobody is using such things in practice.
> _All_ ebuilds in the Portage tree can be successfully parsed with the
> regexp proposed.

I'm not saying they are valid EAPI syntax; but they are all valid
bash. I tend to assume all authors are as...ignorant as myself. Lets
not give them the rope to hang themselves.

>
> The obvious sanity check, i.e. comparing the EAPI obtained by parsing
> and by sourcing, could be added to repoman, which would prevent such
> non-conforming ebuilds from being committed to the tree.
>
>>> *** Proposal 2: "EAPI in header comment" ***
>>> [...]
>
>> Overloading is bad.
>
>> There is no real difference between:
>> #!/usr/bin/ebuild --eapi 5
>> # EAPI=5
>
>> The problem is that the former is also the way to specify how to
>> execute the ebuild; so unless you plan to make ebuilds executable and
>> having /usr/bin/ebuild provide the ebuild environment, using that
>> syntax is confusing to users.
>
> I agree with this point.
>
> Many file formats are identifying themselves with a magic token (as
> it is used by sys-apps/file), but it is not necessarily a shebang.
>
> Ulrich
>

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/08/2012 12:13 AM, Alec Warner wrote:
> On Wed, Mar 7, 2012 at 11:27 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
>> Such constructs also cannot be used with any of the other proposed
>> solutions. And in fact, nobody is using such things in practice.
>> _All_ ebuilds in the Portage tree can be successfully parsed with the
>> regexp proposed.
> 
> I'm not saying they are valid EAPI syntax; but they are all valid
> bash. I tend to assume all authors are as...ignorant as myself. Lets
> not give them the rope to hang themselves.

Something like DEPEND="foo bar" is also valid bash, and yet we don't
allow that either because "foo bar" does not contain valid dependency
atoms. Also, keep in mind that we're not allowing people to "hang
themselves" with invalid EAPI assignments. We'll simply give them a
reasonable error message so that they can quickly and easily correct
their mistake.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[David Leverton]

On Mar 8, 2012 3:29 PM, "Zac Medico" <zmedico@gentoo.org> wrote:
> Something like DEPEND="foo bar" is also valid bash, and yet we don't
> allow that either because "foo bar" does not contain valid dependency
> atoms.

There's a bit of a difference between caring about the value of a
variable and caring about what syntax was used to assign it....

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/08/2012 08:11 AM, David Leverton wrote:
> On Mar 8, 2012 3:29 PM, "Zac Medico"<zmedico@gentoo.org>  wrote:
>> Something like DEPEND="foo bar" is also valid bash, and yet we don't
>> allow that either because "foo bar" does not contain valid dependency
>> atoms.
>
> There's a bit of a difference between caring about the value of a
> variable and caring about what syntax was used to assign it....

Maybe that sort of distinction truly makes a difference to some people, 
but to me it just seems like hair-splitting [1].

[1] http://en.wikipedia.org/wiki/Trivial_objections
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 792 bytes --]

On Thu, 08 Mar 2012 08:21:53 -0800
Zac Medico <zmedico@gentoo.org> wrote:
> Maybe that sort of distinction truly makes a difference to some
> people, but to me it just seems like hair-splitting [1].

So just to get this straight, you think that the following two
restrictions are effectively equivalent?

* The variable DEPEND's value must be set according to the following
  rules:

* The EAPI variable assignment must not use full bash syntax. Instead,
  it must be assigned according to the following rules:

And you believe that having exactly one place inside ebuild text where
there are different whitespace, quoting and indenting rules for
something that otherwise looks exactly like any other metadata variable
isn't going to cause problems?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/08/2012 08:29 AM, Ciaran McCreesh wrote:
> On Thu, 08 Mar 2012 08:21:53 -0800
> Zac Medico<zmedico@gentoo.org>  wrote:
>> Maybe that sort of distinction truly makes a difference to some
>> people, but to me it just seems like hair-splitting [1].
>
> So just to get this straight, you think that the following two
> restrictions are effectively equivalent?
>
> * The variable DEPEND's value must be set according to the following
>    rules:
>
> * The EAPI variable assignment must not use full bash syntax. Instead,
>    it must be assigned according to the following rules:

Yeah, I think they're reasonably/roughly equivalent for how we use them 
in the current context.

> And you believe that having exactly one place inside ebuild text where
> there are different whitespace, quoting and indenting rules for
> something that otherwise looks exactly like any other metadata variable
> isn't going to cause problems?

Yes, with the stipulation that we implement a sanity check / feedback 
mechanism which guarantees that the probed EAPI is identical with the 
result obtained from bash [1].

[1] https://bugs.gentoo.org/show_bug.cgi?id=402167#c18
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alexandre Rostovtsev]

On Thu, 2012-03-08 at 16:29 +0000, Ciaran McCreesh wrote:
> And you believe that having exactly one place inside ebuild text where
> there are different whitespace, quoting and indenting rules for
> something that otherwise looks exactly like any other metadata variable
> isn't going to cause problems?

In light of the fact that all 29758 ebuilds in portage already satisfy
the proposed whitespace, quoting, and indenting constrains on EAPI
assignment, the probability of problems appears to be vanishingly small.
And "vanishingly small" and can be reduced to zero by simply adding a
check to repoman.

Can you come up with a sane scenario under which an ebuild writer would
want to use a different syntax for setting EAPI?

-Alexandre

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

On Thu, 08 Mar 2012 11:59:33 -0500
Alexandre Rostovtsev <tetromino@gentoo.org> wrote:
> On Thu, 2012-03-08 at 16:29 +0000, Ciaran McCreesh wrote:
> > And you believe that having exactly one place inside ebuild text
> > where there are different whitespace, quoting and indenting rules
> > for something that otherwise looks exactly like any other metadata
> > variable isn't going to cause problems?
> 
> In light of the fact that all 29758 ebuilds in portage already satisfy
> the proposed whitespace, quoting, and indenting constrains on EAPI
> assignment, the probability of problems appears to be vanishingly
> small. And "vanishingly small" and can be reduced to zero by simply
> adding a check to repoman.

Because they were recently changed, presumably...

https://bugs.gentoo.org/show_bug.cgi?id=402167#c36

We had this discussion the last time around too, and people were told
to assign in a particular way. As you can see, it didn't work.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ulrich Mueller]

>>>>> On Thu, 8 Mar 2012, Ciaran McCreesh wrote:

> On Thu, 08 Mar 2012 11:59:33 -0500
> Alexandre Rostovtsev <tetromino@gentoo.org> wrote:
>> In light of the fact that all 29758 ebuilds in portage already
>> satisfy the proposed whitespace, quoting, and indenting constrains
>> on EAPI assignment, the probability of problems appears to be
>> vanishingly small. And "vanishingly small" and can be reduced to
>> zero by simply adding a check to repoman.

> Because they were recently changed, presumably...

> https://bugs.gentoo.org/show_bug.cgi?id=402167#c36

> We had this discussion the last time around too, and people were
> told to assign in a particular way. As you can see, it didn't work.

Sorry, but this is nonsense (or rather FUD). Indeed we had 3 ebuilds
(0.01%) in the Portage tree where parsing resulted in an EAPI
different from the one in metadata.

In one of them, removal of the old assignment statement had simply
been forgotten [1]. For the other two, the EAPI had been assigned by
an eclass [2], which we consider illegal anyway.

Ulrich


[1] <http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/dev-ml/bin-prot/bin-prot-2.0.3.ebuild?r1=1.1&r2=1.2>
[2] <http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/eclass/apache-2.eclass?r1=1.26&r2=1.27>

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 471 bytes --]

On Thu, 8 Mar 2012 20:17:41 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> In one of them, removal of the old assignment statement had simply
> been forgotten [1]. For the other two, the EAPI had been assigned by
> an eclass [2], which we consider illegal anyway.

...and yet people do it. That and the violations of the HOMEPAGE rule
tell you a lot about what happens when something is made syntactically
valid but supposedly not legal.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alexis Ballier]

On Thu, 8 Mar 2012 19:31:16 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Thu, 8 Mar 2012 20:17:41 +0100
> Ulrich Mueller <ulm@gentoo.org> wrote:
> > In one of them, removal of the old assignment statement had simply
> > been forgotten [1]. For the other two, the EAPI had been assigned by
> > an eclass [2], which we consider illegal anyway.
> 
> ...and yet people do it. That and the violations of the HOMEPAGE rule
> tell you a lot about what happens when something is made syntactically
> valid but supposedly not legal.
> 

... and this is where repoman helps.
broken deps are syntactically valid but not legal either.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Marc Schiffbauer]

[-- Attachment #1: Type: text/plain, Size: 1229 bytes --]

* Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr:
> >>>>> On Wed, 7 Mar 2012, Alec Warner wrote:
> 
> >> *** Proposal 1: "Parse the EAPI assignment statement" ***
> >> [...]
> 
> > I don't like this idea because the sane way should be easy and
> > straightforward. Mixing a constant declaration with bash assignment
> > just confuses users who think the assignment is full bash when in
> > fact it is not.
> 
> > EAPI=$(somefunc)
> > EAPI=${SOMEVAR%%-*}
> > and so forth all don't meet the regex (and would be flagged
> > invalid.) However a naive author might think they work.
> 
> Such constructs also cannot be used with any of the other proposed
> solutions. And in fact, nobody is using such things in practice.
> _All_ ebuilds in the Portage tree can be successfully parsed with the
> regexp proposed.

Ebuilds are bash scripts. I think introducing exceptions or
constraints here is not straightforward.

I think the only relevant part whether EAPI is set correctly or not
should be the outcome of $EAPI.

I would vote for a solution in a bash comment where repoman would
have to check for its existance and equality to $EAPI.

-Marc
-- 
8AAC 5F46 83B4 DB70 8317  3723 296C 6CCA 35A6 4134

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/08/2012 01:42 AM, Marc Schiffbauer wrote:
> * Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr:
>> Such constructs also cannot be used with any of the other proposed
>> solutions. And in fact, nobody is using such things in practice.
>> _All_ ebuilds in the Portage tree can be successfully parsed with the
>> regexp proposed.
>
> Ebuilds are bash scripts. I think introducing exceptions or
> constraints here is not straightforward.

Given that ebuilds already have to conform to a vast number of 
constraints that ordinary bash scripts do not. I think that it's 
perfectly reasonable for ebuilds to have a constrained syntax for EAPI 
assignments.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

On Thu, 08 Mar 2012 08:30:57 -0800
Zac Medico <zmedico@gentoo.org> wrote:
> On 03/08/2012 01:42 AM, Marc Schiffbauer wrote:
> > * Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr:
> >> Such constructs also cannot be used with any of the other proposed
> >> solutions. And in fact, nobody is using such things in practice.
> >> _All_ ebuilds in the Portage tree can be successfully parsed with
> >> the regexp proposed.
> >
> > Ebuilds are bash scripts. I think introducing exceptions or
> > constraints here is not straightforward.
> 
> Given that ebuilds already have to conform to a vast number of 
> constraints that ordinary bash scripts do not. I think that it's 
> perfectly reasonable for ebuilds to have a constrained syntax for
> EAPI assignments.

...and only EAPI assignments? Not for any other metadata variable?

Doesn't that sort of suggest that EAPI shouldn't be a metadata variable?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/08/2012 08:35 AM, Ciaran McCreesh wrote:
> On Thu, 08 Mar 2012 08:30:57 -0800
> Zac Medico<zmedico@gentoo.org>  wrote:
>> On 03/08/2012 01:42 AM, Marc Schiffbauer wrote:
>>> * Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr:
>>>> Such constructs also cannot be used with any of the other proposed
>>>> solutions. And in fact, nobody is using such things in practice.
>>>> _All_ ebuilds in the Portage tree can be successfully parsed with
>>>> the regexp proposed.
>>>
>>> Ebuilds are bash scripts. I think introducing exceptions or
>>> constraints here is not straightforward.
>>
>> Given that ebuilds already have to conform to a vast number of
>> constraints that ordinary bash scripts do not. I think that it's
>> perfectly reasonable for ebuilds to have a constrained syntax for
>> EAPI assignments.
>
> ...and only EAPI assignments? Not for any other metadata variable?

It's only needed for the EAPI, since that's the only value defined by 
the ebuild that we intend to use to control how the global environment 
is initialized prior to sourcing of the ebuild.

> Doesn't that sort of suggest that EAPI shouldn't be a metadata variable?

It's a very special metadata variable. Of course, it could also be 
implemented in many different ways that do not involve bash variable 
assingments. Maybe the differences between the various possible ways 
truly make a difference to some people, but to me it's just 
hair-splitting [1].

[1] http://en.wikipedia.org/wiki/Trivial_objections
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

On Thu, 08 Mar 2012 09:07:18 -0800
Zac Medico <zmedico@gentoo.org> wrote:
> It's a very special metadata variable. Of course, it could also be 
> implemented in many different ways that do not involve bash variable 
> assingments. Maybe the differences between the various possible ways 
> truly make a difference to some people, but to me it's just 
> hair-splitting [1].
> 
> [1] http://en.wikipedia.org/wiki/Trivial_objections

Having a different, special rule for something that looks exactly like
lots of other things that do not have that different, special rule is
hardly hair splitting. This rule would have to be documented and have
special code to carefully enforce it. That's a big deal.

Having something break because you add an unrelated comment to the top
of a file is weird.

Having something break because you indent it, where nothing else breaks
if you do the same thing, is weird.

Having something break because you make full use of bash syntax, where
nothing else breaks if you do the same thing, is weird.

There are already a whole pile of subtle traps for ebuild writers and
complications for people learning the system. We should be aiming to
reduce these, not add to them.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Jeroen Roovers]

On Thu, 8 Mar 2012 17:14:58 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> Having a different, special rule for something that looks exactly like
> lots of other things that do not have that different, special rule is
> hardly hair splitting. This rule would have to be documented and have
> special code to carefully enforce it. That's a big deal.

Exactly like HOMEPAGE is a big deal?

HOMEPAGE	 Package's homepage. If you are unable to locate an
official one, try to provide a link to freshmeat.net or a similar
package tracking site. Never refer to a variable name in the string;
include only raw text.

http://devmanual.gentoo.org/ebuild-writing/variables/index.html


   jer

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

On Thu, 8 Mar 2012 18:30:47 +0100
Jeroen Roovers <jer@gentoo.org> wrote:
> On Thu, 8 Mar 2012 17:14:58 +0000
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > Having a different, special rule for something that looks exactly
> > like lots of other things that do not have that different, special
> > rule is hardly hair splitting. This rule would have to be
> > documented and have special code to carefully enforce it. That's a
> > big deal.
> 
> Exactly like HOMEPAGE is a big deal?
> 
> HOMEPAGE	 Package's homepage. If you are unable to locate an
> official one, try to provide a link to freshmeat.net or a similar
> package tracking site. Never refer to a variable name in the string;
> include only raw text.
> 
> http://devmanual.gentoo.org/ebuild-writing/variables/index.html

That's a QA thing, not a spec thing, and it's there because some
old-school developers don't know how to use their package manager
properly, and want to copy-paste a string from an ebuild into their
browser.

It's a silly rule. And if you take a count of how many ebuilds and
eclasses break that rule, you'll see exactly why such rules are a
problem.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Kent Fredric]

On 9 March 2012 06:30, Jeroen Roovers <jer@gentoo.org> wrote:
> On Thu, 8 Mar 2012 17:14:58 +0000
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
>
>> Having a different, special rule for something that looks exactly like
>> lots of other things that do not have that different, special rule is
>> hardly hair splitting. This rule would have to be documented and have
>> special code to carefully enforce it. That's a big deal.
>
> Exactly like HOMEPAGE is a big deal?
>
> HOMEPAGE         Package's homepage. If you are unable to locate an
> official one, try to provide a link to freshmeat.net or a similar
> package tracking site. Never refer to a variable name in the string;
> include only raw text.

I'm not sure if that is a good example of "things that are weird like EAPI"

You can set HOMEPAGE in an eclass for instance, using anything you can
do with bash and build it out of substrings.

perl-module.eclass does just that.

[[ -z "${HOMEPAGE}" ]] && \
        HOMEPAGE="http://search.cpan.org/dist/${MY_PN:-${PN}}/"

Not really comparable in entirety to EAPI.

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

On Sat, 10 Mar 2012 14:06:49 +1300
Kent Fredric <kentfredric@gmail.com> wrote:
> > HOMEPAGE         Package's homepage. If you are unable to locate an
> > official one, try to provide a link to freshmeat.net or a similar
> > package tracking site. Never refer to a variable name in the string;
> > include only raw text.
> 
> I'm not sure if that is a good example of "things that are weird like
> EAPI"
> 
> You can set HOMEPAGE in an eclass for instance, using anything you can
> do with bash and build it out of substrings.
> 
> perl-module.eclass does just that.
> 
> [[ -z "${HOMEPAGE}" ]] && \
>         HOMEPAGE="http://search.cpan.org/dist/${MY_PN:-${PN}}/"
> 
> Not really comparable in entirety to EAPI.

The point is that there's a silly QA policy saying not to do that. It's
not a PMS rule.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Marc Schiffbauer]

[-- Attachment #1: Type: text/plain, Size: 1473 bytes --]

* Zac Medico schrieb am 08.03.12 um 17:30 Uhr:
> On 03/08/2012 01:42 AM, Marc Schiffbauer wrote:
> > * Ulrich Mueller schrieb am 08.03.12 um 08:27 Uhr:
> >> Such constructs also cannot be used with any of the other proposed
> >> solutions. And in fact, nobody is using such things in practice.
> >> _All_ ebuilds in the Portage tree can be successfully parsed with the
> >> regexp proposed.
> >
> > Ebuilds are bash scripts. I think introducing exceptions or
> > constraints here is not straightforward.
> 
> Given that ebuilds already have to conform to a vast number of 
> constraints that ordinary bash scripts do not. I think that it's 
> perfectly reasonable for ebuilds to have a constrained syntax for EAPI 
> assignments.

There are constraints in ebuilds, right. But its an *ebuild* in the
end, not an ordinary shell script. Thats true.

So if EAPI is very special, and I am now convinced it is, then well, 
this might be the most important contraint in an ebuild at all.

If that is true I would vote to keep this as simple as possible:

* EAPI *must* *be* the first non-Argument / shell code in an ebuild
* The value of EAPI in the assignment *MUST* *NOT* contain any
  other variables or other shell substitutions.

Period. Done.

* That would be the least invasive change.
* Could easily be checked by repoman
* Is easy parsable by other programs (python code)

-Marc
-- 
8AAC 5F46 83B4 DB70 8317  3723 296C 6CCA 35A6 4134

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3932 bytes --]

On Wed, 7 Mar 2012 20:12:25 -0800
Alec Warner <antarus@gentoo.org> wrote:

> On Wed, Mar 7, 2012 at 12:41 PM, Ulrich Mueller <ulm@gentoo.org>
> wrote:
> > Hi all,
> >
> > The way how we currently specify the EAPI in ebuilds has some
> > problems. For example, there is no sane way to allow usage of
> > features of a new bash version in a new EAPI. So we are currently
> > stuck with bash 3.2. Also changes of global scope behaviour, like
> > addition of new global scope functions (similar to "inherit") are
> > not possible.
> >
> > These flaws are outlined in GLEP 55 [1]:
> > | In order to get the EAPI the package manager needs to source the
> > | ebuild, which itself needs the EAPI in the first place. Otherwise
> > it | imposes a serious limitation, namely every ebuild, using any
> > of the | future EAPIs, will have to be source'able by old package
> > managers | [...]
> >
> > The council has voted down GLEP 55 more than a year ago, but at the
> > same time requested that a solution for the mentioned issues should
> > be found. [2] However, there was no progress since then.
> >
> > The issue arose again in bug 402167 [3] where several solutions have
> > been discussed. Below, I try to summarise the possible options
> > resulting from that discussion.
> >
> >
> > *** Proposal 1: "Parse the EAPI assignment statement" ***
> >
> > This first proposal would require that the syntax of the EAPI
> > assignment statement in ebuilds matches a well defined regular
> > expression. A scan of the Portage tree shows that the statement only
> > occurs in the following variations (using EAPI 4 as example):
> >
> >   EAPI=4
> >   EAPI="4"
> >   EAPI='4'
> >
> > Sometimes this is followed by whitespace or a comment (starting with
> > a # sign). Also, with very few exceptions the EAPI assignment occurs
> > within the first few lines of the ebuild. For the vast majority of
> > ebuilds it is in line 5.
> >
> > Written in a more formal way, appropriate for a specification:
> > - Ebuilds must contain at most one EAPI assignment statement.
> > - It must occur within the first N lines of the ebuild (N=10 and
> > N=30 have been suggested).
> > - The statement must match the following regular expression
> > (extended regexp syntax):
> >  ^[ \t]*EAPI=(['"]?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$
> >
> > Note: The first and the third point are already fulfilled by all
> > ebuilds in the Portage tree. The second point will require very few
> > ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).
> >
> > The package manager would determine the EAPI by parsing the
> > assignment with above regular expression. A sanity check would be
> > added. Citing Zac Medico in [3]: "The fact that we can compare the
> > probed EAPI to the actual EAPI variable after the ebuild is sourced
> > seems like a perfect sanity check. We could easily detect
> > inconsistencies and flag such ebuilds as invalid, providing a
> > reliable feedback mechanism to ebuild developers."
> >
> > This proposal comes in two variants:
> > 1a) The change is applied retroactively for all EAPIs.
> > 1b) It is only applied for EAPI 5 and later (which means that the
> >    result of the EAPI parsing would be discarded for earlier EAPIs).
> 
> I don't like this idea because the sane way should be easy and
> straightforward. Mixing a constant declaration with bash assignment
> just confuses users who think the assignment is full bash when in
> fact it is not.
> 
> EAPI=$(somefunc)
> EAPI=${SOMEVAR%%-*}
> and so forth all don't meet the regex (and would be flagged invalid.)
> However a naive author might think they work.

And they all should be invalid due to our policies. The most important
ebuild variables like EAPI should be readable on sight, without having
to lookup random variables, functions etc.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michael Orlitzky]

On 03/07/2012 03:41 PM, Ulrich Mueller wrote:
>
> *** Proposal 1: "Parse the EAPI assignment statement" ***
>

There's also libbash now:

   http://www.gentoo.org/proj/en/libbash/index.xml

Anyone know how close we are to being able to use it to parse the EAPI?

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ulrich Mueller]

>>>>> On Thu, 08 Mar 2012, Michael Orlitzky wrote:

> On 03/07/2012 03:41 PM, Ulrich Mueller wrote:
>> 
>> *** Proposal 1: "Parse the EAPI assignment statement" ***

> There's also libbash now:

>    http://www.gentoo.org/proj/en/libbash/index.xml

Looks like complete overkill to me, considering the simple task at
hand.

Ulrich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Rich Freeman]

On Thu, Mar 8, 2012 at 11:51 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Thu, 08 Mar 2012, Michael Orlitzky wrote:
>
>> There's also libbash now:
>
> Looks like complete overkill to me, considering the simple task at
> hand.
>

Plus, wasn't the whole point that we can't guarantee that the bash
installed on a user's system can parse the ebuild until we've checked
the EAPI?  If we use libbash doesn't that just keep the same
constraint but on a different package?  Is the current stable libbash
guaranteed to be able to parse a bash v12 script?

If you just parse the file with a defined set of rules without regard
to how bash might parse the file, then you can determine the EAPI and
then decide how to source it.  For all we know EAPI G will turn
ebuilds into python scripts so trying to read the thing with bash or
libbash will be doomed to failure.

Rich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Mike Gilbert]

On Wed, Mar 7, 2012 at 3:41 PM, Ulrich Mueller <ulm@gentoo.org> wrote:
> Again, the proposal comes in two variants:
> 2a) It is combined with a one time change of the file extension, like
>    .ebuild -> .eb.
> 2b) The usual EAPI assignment statement in the ebuild is still
>    required, at least for a transition period.
>

Just throwing my opinion in:

I like proposal 2 better than proposal 1. Placing a regex-based
constraint on a bash variable assignment doesn't feel right to me.

I slightly prefer 2a over 2b because it feels cleaner. I'm sure it
will break some tools, but I don't have a good feel for the scope of
that breakage.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2315 bytes --]

On Wed, 7 Mar 2012 21:41:02 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> *** Proposal 1: "Parse the EAPI assignment statement" ***
>
> [...]
> 
> Written in a more formal way, appropriate for a specification:
> - Ebuilds must contain at most one EAPI assignment statement.
> - It must occur within the first N lines of the ebuild (N=10 and N=30
>   have been suggested).
> - The statement must match the following regular expression (extended
>   regexp syntax):
>   ^[ \t]*EAPI=(['"]?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$

I'd make the regexp less strict -- at least allow whitespace around '='.
If the intent is to not rely on a specific bash version for a global
scope, why should we limit it to the (current) bash syntax?

And it may be also a good idea to not rely on a specific line format,
so e.g. 'dnl EAPI=4' will work as well.

> 1b) It is only applied for EAPI 5 and later (which means that the
>     result of the EAPI parsing would be discarded for earlier EAPIs).

Err... so what happens if 'new parsing' detects EAPI 4 and 'old
parsing' detects EAPI 5? Or more exactly, how does it know when
an older EAPI is used if it is supposed to not use the value it uses to
detect EAPI?

> *** Proposal 2: "EAPI in header comment" ***
> 
> A different approach would be to specify the EAPI in a specially
> formatted comment in the ebuild's header. No syntax has been suggested
> yet, but I believe that the following would work as a specification:
> - The EAPI must be declared in a special comment in the first line of
>   the ebuild's header, as follows:
> - The first line of the ebuild must contain the word "ebuild",
>   followed by whitespace, followed by the EAPI, followed by
>   end-of-line or whitespace.

What if we ever decide to use a language which would would have another
requirements for first line?

> Again, the proposal comes in two variants:
> 2a) It is combined with a one time change of the file extension, like
>     .ebuild -> .eb.

And we're going to retroactively migrate the tree or have random file
suffixes intermixed? Not to mention we're either keeping two different
variants for a longer while, or disregarding backwards compatibility
with older package managers for no actual benefit.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 689 bytes --]

On Thu, 8 Mar 2012 18:52:13 +0100
Michał Górny <mgorny@gentoo.org> wrote:
> > Again, the proposal comes in two variants:
> > 2a) It is combined with a one time change of the file extension,
> > like .ebuild -> .eb.
> 
> And we're going to retroactively migrate the tree or have random file
> suffixes intermixed? Not to mention we're either keeping two different
> variants for a longer while, or disregarding backwards compatibility
> with older package managers for no actual benefit.

No, you'd just switch extensions for EAPI 5 onwards, and keep the old
rules for EAPI 4. Same idea and impact as GLEP 55, except without as
much future proofing.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Zac Medico]

On 03/08/2012 09:52 AM, Michał Górny wrote:
> On Wed, 7 Mar 2012 21:41:02 +0100
> Ulrich Mueller<ulm@gentoo.org>  wrote:
>> 1b) It is only applied for EAPI 5 and later (which means that the
>>      result of the EAPI parsing would be discarded for earlier EAPIs).
>
> Err... so what happens if 'new parsing' detects EAPI 4 and 'old
> parsing' detects EAPI 5? Or more exactly, how does it know when
> an older EAPI is used if it is supposed to not use the value it uses to
> detect EAPI?

We can simply detect this case and treat it as an error. The purpose of 
the "discarded for earlier EAPIs" part is to allow more variance for 
older EAPIs, and treating this case as an error will probably affect 
zero or a negligible number of older ebuilds in practice.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ulrich Mueller]

>>>>> On Thu, 8 Mar 2012, Michał Górny wrote:

>> *** Proposal 1: "Parse the EAPI assignment statement" ***
>> 
>> [...]
>> 
>> Written in a more formal way, appropriate for a specification:
>> - Ebuilds must contain at most one EAPI assignment statement.
>> - It must occur within the first N lines of the ebuild (N=10 and N=30
>> have been suggested).
>> - The statement must match the following regular expression (extended
>> regexp syntax):
>> ^[ \t]*EAPI=(['"]?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$

> I'd make the regexp less strict -- at least allow whitespace around '='.
> If the intent is to not rely on a specific bash version for a global
> scope, why should we limit it to the (current) bash syntax?

This could certainly be done ...

> And it may be also a good idea to not rely on a specific line format,
> so e.g. 'dnl EAPI=4' will work as well.

... but loosening it too much looks a little dangerous, because it
would probably match comments and could produce unintended matches.

If we allow for a general syntax, we should restrict it to the first
line of the ebuild. But that is proposal 2 then.

>> 1b) It is only applied for EAPI 5 and later (which means that the
>> result of the EAPI parsing would be discarded for earlier EAPIs).

> Err... so what happens if 'new parsing' detects EAPI 4 and 'old
> parsing' detects EAPI 5?

This cannot happen for a legal ebuild:
- If the ebuild is EAPI 4, then sourcing it ("old parsing") must
  detect EAPI 4.
- If the ebuild is EAPI 5, then "new parsing" must detect EAPI 5.

> Or more exactly, how does it know when an older EAPI is used if it
> is supposed to not use the value it uses to detect EAPI?

If new parsing detects EAPI 5 or later, then use this value (and do a
sanity check after the ebuild has been sourced). Otherwise, discard
the value determined by new parsing and detect the EAPI in the
traditional way by sourcing the ebuild.

Ulrich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Alexis Ballier]

On Thu, 8 Mar 2012 20:04:55 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:
> > Err... so what happens if 'new parsing' detects EAPI 4 and 'old
> > parsing' detects EAPI 5?
> 
> This cannot happen for a legal ebuild:
> - If the ebuild is EAPI 4, then sourcing it ("old parsing") must
>   detect EAPI 4.

the problem here may be when "old parsing" will wrongly parse a
malformed EAPI 42 assignment (so that new parsing doesnt detect an
EAPI and assumes its 0) as EAPI 4.
this is a purely hypothetical scenario that will be easily detected if
it ever happens, though :)

anyway, "new parsing" != "old parsing" should trigger a repoman
warning for old EAPIs.

A.

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Walter Dnes]

On Wed, Mar 07, 2012 at 09:41:02PM +0100, Ulrich Mueller wrote

> Written in a more formal way, appropriate for a specification:
> - Ebuilds must contain at most one EAPI assignment statement.
> - It must occur within the first N lines of the ebuild (N=10 and N=30
>   have been suggested).
> - The statement must match the following regular expression (extended
>   regexp syntax):
>   ^[ \t]*EAPI=(['"]?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$
> 
> Note: The first and the third point are already fulfilled by all
> ebuilds in the Portage tree. The second point will require very few
> ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).

  The second point could be rendered moot with a little sanity checking
like so...

#!/bin/bash
counter=`grep -c "^EAPI=" ${1}`
if [ ${counter} -eq 0 ]; then
#  <call routine for processing EAPI 0>
   exit
elif [ ${counter} -gt 1 ]; then
   echo "***ERROR*** only 1 line allowed beginning with EAPI="
   exit
else
#  <parse value of EAPI and call appropriate ebuild parser version>
   exit
fi

  This treats the EAPI statement as a declaration, and doesn't care
where it shows up in the file.  I'm assuming no leading blanks in front
of "EAPI".

-- 
Walter Dnes <waltdnes@waltdnes.org>

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ralph Sennhauser]

[-- Attachment #1: Type: text/plain, Size: 7157 bytes --]

On Wed, 7 Mar 2012 21:41:02 +0100
Ulrich Mueller <ulm@gentoo.org> wrote:

> Hi all,
> 
> The way how we currently specify the EAPI in ebuilds has some
> problems. For example, there is no sane way to allow usage of features
> of a new bash version in a new EAPI. So we are currently stuck with
> bash 3.2. Also changes of global scope behaviour, like addition of new
> global scope functions (similar to "inherit") are not possible.
> 
> These flaws are outlined in GLEP 55 [1]:
> | In order to get the EAPI the package manager needs to source the
> | ebuild, which itself needs the EAPI in the first place. Otherwise it
> | imposes a serious limitation, namely every ebuild, using any of the
> | future EAPIs, will have to be source'able by old package managers
> | [...]
> 
> The council has voted down GLEP 55 more than a year ago, but at the
> same time requested that a solution for the mentioned issues should be
> found. [2] However, there was no progress since then.
> 
> The issue arose again in bug 402167 [3] where several solutions have
> been discussed. Below, I try to summarise the possible options
> resulting from that discussion.
> 
> 
> *** Proposal 1: "Parse the EAPI assignment statement" ***
> 
> This first proposal would require that the syntax of the EAPI
> assignment statement in ebuilds matches a well defined regular
> expression. A scan of the Portage tree shows that the statement only
> occurs in the following variations (using EAPI 4 as example):
> 
>    EAPI=4
>    EAPI="4"
>    EAPI='4'
> 
> Sometimes this is followed by whitespace or a comment (starting with
> a # sign). Also, with very few exceptions the EAPI assignment occurs
> within the first few lines of the ebuild. For the vast majority of
> ebuilds it is in line 5.
> 
> Written in a more formal way, appropriate for a specification:
> - Ebuilds must contain at most one EAPI assignment statement.
> - It must occur within the first N lines of the ebuild (N=10 and N=30
>   have been suggested).
> - The statement must match the following regular expression (extended
>   regexp syntax):
>   ^[ \t]*EAPI=(['"]?)([A-Za-z0-9._+-]*)\1[ \t]*(#.*)?$
> 
> Note: The first and the third point are already fulfilled by all
> ebuilds in the Portage tree. The second point will require very few
> ebuilds to be changed (9 packages for N=10, or 2 packages for N=30).
> 
> The package manager would determine the EAPI by parsing the assignment
> with above regular expression. A sanity check would be added. Citing
> Zac Medico in [3]: "The fact that we can compare the probed EAPI to
> the actual EAPI variable after the ebuild is sourced seems like a
> perfect sanity check. We could easily detect inconsistencies and flag
> such ebuilds as invalid, providing a reliable feedback mechanism to
> ebuild developers."
> 
> This proposal comes in two variants:
> 1a) The change is applied retroactively for all EAPIs.
> 1b) It is only applied for EAPI 5 and later (which means that the
>     result of the EAPI parsing would be discarded for earlier EAPIs).
> 
> 
> *** Proposal 2: "EAPI in header comment" ***
> 
> A different approach would be to specify the EAPI in a specially
> formatted comment in the ebuild's header. No syntax has been suggested
> yet, but I believe that the following would work as a specification:
> - The EAPI must be declared in a special comment in the first line of
>   the ebuild's header, as follows:
> - The first line of the ebuild must contain the word "ebuild",
>   followed by whitespace, followed by the EAPI, followed by
>   end-of-line or whitespace.
> 
> Again, the proposal comes in two variants:
> 2a) It is combined with a one time change of the file extension, like
>     .ebuild -> .eb.
> 2b) The usual EAPI assignment statement in the ebuild is still
>     required, at least for a transition period.
> 
> In the 2a case, the EAPI variable could be made read-only in bash
> before sourcing the ebuild. In the 2b case, a sanity check similar to
> the one mentioned above would be added.
> 
> 
> What do you think?
> 
> (I really hope for a constructive discussion here. So, if you want
> to comment that all of the above proposals suck and GLEP 55 is much
> superior, then please open a new thread for it.)
> 
> Ulrich
> 

Currently 5 proposals are listed on the wiki. [4]

While all of them have some temptations the actual goal is to make
obtaining the EAPI the very first step so everything else can be
defined in terms of EAPI and so immediately deployable in future. This
are changes in atom syntax like needed for GLEP 54 or those bash
feature often mentioned besides many other things one can think of. 

GLEP 55 requires changing ebuild extensions on a regular basis but
doesn't impose any limit on the ebuild format or atom syntax, only the
file extensions would be imposed. The ebuild extensions for GLEP 55
would likely always be ebuild-<integer> as integers are reserved for
future use by Gentoo. While for example .ebuild-5 is still recognised
as an ebuild; .eb .ebld .ebd .bld .dliube .dlbe .be are not. This
brings me to the point if not GLEP 55 then only if we can keep .ebuild.
That one-time extension change thingy smells like being stubborn.

If we aren't going to to change the extension the most sensible thing
is to pack the eapi into the ebuild as done now but get rid of as many
limitations as possible. Looking at the proposals for eapi specification
in ebuilds the header comment is the clear winner here. The eapi can
also instruct the package manager to remove the first line for further
use of the ebuild if required for obscure formats / languages.

I already ruled out proposal "3 EAPI in header comment and one-time
change of file extension" for me as an option, see above. And "2 EAPI
in header comment" comes with compatibility issues for an undefined
amount of time.
If we want to keep .ebuild but avoid the compat issue another variant
would be "EAPI in header comment and one-time change of ebuild
location" or more formal:

6 EAPI in header comment and one-time change of ebuild location:

- add a directory $CATEGORY/$PN/ebuilds to ebuild repositories.
- all files in $CATEGORY/$PN/ebuilds are ebuilds and are using a well
  defined first line to denote the EAPI.
- For practical reasons the header should be a bash comment. PMs
  shouldn't have to remove or skip first line from file for further
  processing of ebuilds supporting bash comments.
- the .ebuild extension can be kept but could be changed if ever
  desired. This due to the filename only having meaning if the
  EAPI of the file is known.

Comparing this with GLEP 55 then this allows us to keep .ebuild in
return of some overhead with roughly the same pros and cons otherwise,
right?

Ralph


> 
> [1] <http://www.gentoo.org/proj/en/glep/glep-0055.html#problem>
> [2]
> <http://www.gentoo.org/proj/en/council/meeting-logs/20100823-summary.txt>
> [3] <https://bugs.gentoo.org/show_bug.cgi?id=402167>
> 
[4] <http://wiki.gentoo.org/wiki/Alternate_EAPI_mechanisms>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Rich Freeman]

On Sun, Mar 18, 2012 at 3:23 AM, Ralph Sennhauser <sera@gentoo.org> wrote:
> The ebuild extensions for GLEP 55
> would likely always be ebuild-<integer> as integers are reserved for
> future use by Gentoo.

Looking at GLEP 55 the proposal was ebuild-<EAPI> - not
ebuild-<integer>.  I don't believe there is any restriction that EAPIs
be integers or incremental in nature.

Don't mean to nitpick, but perhaps not all realize this...

Rich

Re: [gentoo-dev] RFD: EAPI specification in ebuilds

[Ulrich Mueller]

>>>>> On Sun, 18 Mar 2012, Ralph Sennhauser wrote:

> If we want to keep .ebuild but avoid the compat issue another
> variant would be "EAPI in header comment and one-time change of
> ebuild location" or more formal:

> 6 EAPI in header comment and one-time change of ebuild location:

> - add a directory $CATEGORY/$PN/ebuilds to ebuild repositories.
> - all files in $CATEGORY/$PN/ebuilds are ebuilds and are using a
>   well defined first line to denote the EAPI.
> - For practical reasons the header should be a bash comment. PMs
>   shouldn't have to remove or skip first line from file for further
>   processing of ebuilds supporting bash comments.
> - the .ebuild extension can be kept but could be changed if ever
>   desired. This due to the filename only having meaning if the
>   EAPI of the file is known.

Similar ideas have been discussed before, in 2007 and 2009:
<http://archives.gentoo.org/gentoo-dev/msg_fe3f0b9d050ead86aed9b42ce7ec93b0.xml>
<http://archives.gentoo.org/gentoo-dev/msg_2e41942be33d8595cf7152aa91417fbe.xml>

> Comparing this with GLEP 55 then this allows us to keep .ebuild in
> return of some overhead with roughly the same pros and cons
> otherwise, right?

From a technical point of view, it's the same pros and cons. There are
however non-technical aspects for all these propositions. For example,
we would have to live with ebuilds in different directories for a long
transition period (as we would have to live with two file extensions
for a long time if we changed that).

Ulrich

[gentoo-dev] Re: RFD: EAPI specification in ebuilds

[Ulrich Mueller]

There is consensus in the gentoo-pms mailing list that we should
tighten the rules for the position of the EAPI assignment in ebuilds,
namely that it should take place in the first non-blank and
non-comment line.

The devmanual already requires that "if you want to override the EAPI
variable, you have to specify it at the top of the ebuild."

In any case, the portage tree mostly complies with this. There are 130
ebuilds in 66 packages (or 0.4 % of the total number) that don't, but
they can be trivially fixed (and I volunteer to do the work).

Any objections against this?

Ulrich

Re: [gentoo-dev] Re: RFD: EAPI specification in ebuilds

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 655 bytes --]

On Thursday 12 April 2012 15:53:07 Ulrich Mueller wrote:
> There is consensus in the gentoo-pms mailing list that we should
> tighten the rules for the position of the EAPI assignment in ebuilds,
> namely that it should take place in the first non-blank and
> non-comment line.
> 
> The devmanual already requires that "if you want to override the EAPI
> variable, you have to specify it at the top of the ebuild."
> 
> In any case, the portage tree mostly complies with this. There are 130
> ebuilds in 66 packages (or 0.4 % of the total number) that don't, but
> they can be trivially fixed (and I volunteer to do the work).

do it
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]