[gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ciaran McCreesh wrote:
> Because your proposal addresses none of the underlying problems which
> GLEP 55 was created to solve.

As said long ago the glep doesn't tell enough:

"The current way of specifying the EAPI in ebuilds is flawed. In order 
to get the EAPI the package manager needs to source the ebuild, which 
itself needs the EAPI in the first place. Otherwise it imposes a serious 
limitation, namely every ebuild, using any of the future EAPIs, will 
have to be source'able by old package managers and hence there is no way 
to do any of the following:

         * Change the behaviour of inherit in any way (for example, to 
extend or change eclass functionality).
         * Add new global scope functions in any sane way.
         * Extend versioning rules in an EAPI - for example, addition of 
the scm suffix - GLEP54"

Let's try to start with a common workflow for the user:
- an user with an ancient version of portage syncs
- it requires a package
- it looks at the cache ($portdir/metadata/cache/)
- picks the best entry from the ones showing an eapi it understands
- keeps going.

Apparently we do not have any issue...

Now problems:
1- the cache could get a non compatible change
2- the user triggers a metadata cache regeneration
    -> the ebuild is sourced -> portage could fail or do something 
unpredictable
3- overlays do not provide metadata cache
4- A package manager different from portage do not use the provided cache.

Solutions:
1- move the incompatible cache out of ancient portage scope (like in a 
separate directory)
2- The user will get unpredictable behavior, but portage tell you when 
upgrading is needed...
3- you'd have to disable them
4- unsupported.

Apparently for this side we don't have much to do if we get a valid cache.

Ebuilds have to be added to portage so here the workflow for the developer:

- new ebuild is sourced
- cache is generated
- manifest is built

In this case we have a problem if the source step is a single one, 
portage won't know in advance how to behave.

So the first step has to be split in two:
- first portage discovers which is the eapi version
- then behave as defined by the eapi

The problem is that right now sourcing is done by having an instructed 
bash. So the simplest way to get the first step done is parsing the 
ebuild file with something different like file(1) and then instruct bash 
and do the parsing.

This will solve the issue for the developer.

What is proposed in glep-55 seems to aim to solve both issues at the 
same time (it isn't stated) by switching file extension every time the 
eapi is changed. This is slightly against the principle of the least 
surprise and apparently is disliked by enough people to lead the 
situation to be discussed in the council.

The fact the glep itself is too much terse doesn't help acknowledging 
the problems it aims to solve and the fact it fails to state actual 
issues that may need a solution doesn't make it worth the effort and 
disruption it would lead.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

[gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Tiziano Müller]

> What is proposed in glep-55 seems to aim to solve both issues at the 
> same time (it isn't stated) by switching file extension every time the 
> eapi is changed. This is slightly against the principle of the least 
> surprise and apparently is disliked by enough people to lead the 
> situation to be discussed in the council.
> 

Instead of switching file extension every time the eapi is changed you
could also increment it only when a new EAPI breaks sourcing the ebuild
compared to the requirements of the prior EAPI.
(This way you'd in fact split EAPI into a major- and a minor-version.)

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On Mon, Feb 23, 2009 at 09:38:06AM +0100, Tiziano M??ller wrote:
> > What is proposed in glep-55 seems to aim to solve both issues at the 
> > same time (it isn't stated) by switching file extension every time the 
> > eapi is changed. This is slightly against the principle of the least 
> > surprise and apparently is disliked by enough people to lead the 
> > situation to be discussed in the council.
> > 
> 
> Instead of switching file extension every time the eapi is changed you
> could also increment it only when a new EAPI breaks sourcing the ebuild
> compared to the requirements of the prior EAPI.
> (This way you'd in fact split EAPI into a major- and a minor-version.)

Complicates the implementation (annoying), but more importantly 
negates one of the features of g55- being able to tell via the 
extension if the manager supports that EAPI.

Honestly, the issue isn't breaking sourcing (literally unable to 
source it) of the ebuild as much as sourcing it *wrong*- simplest 
example, new metadata key is added in eapi 1.3, compliant 
implementations have to pull that key out of the env and put it in the 
cache.  Sourcing on the surface, would succeed- but the results would 
be worthless (it'd basically be similar to the situation now).

~brian

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alistair Bush]

Tiziano Müller wrote:
>> What is proposed in glep-55 seems to aim to solve both issues at the 
>> same time (it isn't stated) by switching file extension every time the 
>> eapi is changed. This is slightly against the principle of the least 
>> surprise and apparently is disliked by enough people to lead the 
>> situation to be discussed in the council.
>>
> 
> Instead of switching file extension every time the eapi is changed you
> could also increment it only when a new EAPI breaks sourcing the ebuild
> compared to the requirements of the prior EAPI.
> (This way you'd in fact split EAPI into a major- and a minor-version.)
> 

Doesn't that just add extra complexity for no gain.

Personally I don't see what the problem is with simply implementing
GLEP-55.  It's the best solution.
It should be pretty simple to implement too.  Certainly it wouldn't be
anymore difficult to implement than your solution.

Maybe once zmedico finishes his latest development push I will attempt
to implement it.

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

Am Montag, den 23.02.2009, 22:25 +1300 schrieb Alistair Bush:
> 
> Tiziano Müller wrote:
> >> What is proposed in glep-55 seems to aim to solve both issues at the 
> >> same time (it isn't stated) by switching file extension every time the 
> >> eapi is changed. This is slightly against the principle of the least 
> >> surprise and apparently is disliked by enough people to lead the 
> >> situation to be discussed in the council.
> >>
> > 
> > Instead of switching file extension every time the eapi is changed you
> > could also increment it only when a new EAPI breaks sourcing the ebuild
> > compared to the requirements of the prior EAPI.
> > (This way you'd in fact split EAPI into a major- and a minor-version.)
> > 
> 
> Doesn't that just add extra complexity for no gain.
Yes, sure. I was just looking for a solution for the "we have countless .eapi-X after 10 years" problem.

> Personally I don't see what the problem is with simply implementing
> GLEP-55.  It's the best solution.
> It should be pretty simple to implement too.  Certainly it wouldn't be
> anymore difficult to implement than your solution.

I fully agree.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Douglas Anderson]

On Mon, Feb 23, 2009 at 7:02 PM, Tiziano Müller <dev-zero@gentoo.org> wrote:
> Am Montag, den 23.02.2009, 22:25 +1300 schrieb Alistair Bush:
>>
>> Tiziano Müller wrote:
>> >> What is proposed in glep-55 seems to aim to solve both issues at the
>> >> same time (it isn't stated) by switching file extension every time the
>> >> eapi is changed. This is slightly against the principle of the least
>> >> surprise and apparently is disliked by enough people to lead the
>> >> situation to be discussed in the council.
>> >>
>> >
>> > Instead of switching file extension every time the eapi is changed you
>> > could also increment it only when a new EAPI breaks sourcing the ebuild
>> > compared to the requirements of the prior EAPI.
>> > (This way you'd in fact split EAPI into a major- and a minor-version.)
>> >
>>
>> Doesn't that just add extra complexity for no gain.
> Yes, sure. I was just looking for a solution for the "we have countless .eapi-X after 10 years" problem.

No one wants to be working with ebuild-29 or something like that in a
few years and trying to figure out which feature came in which EAPI.
Instead of bumping EAPI for each little change, save them up and bump
no more than once a year or less, each bump bringing in some major new
feature. With a little common sense and planning, we could make this a
non-issue and give ebuild authors and PM devs alike a little time to
get used to each change.

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2082 bytes --]

On Mon, Feb 23, 2009 at 07:28:00PM +0900, Douglas Anderson wrote:
> On Mon, Feb 23, 2009 at 7:02 PM, Tiziano Müller <dev-zero@gentoo.org> wrote:
> > Am Montag, den 23.02.2009, 22:25 +1300 schrieb Alistair Bush:
> >>
> >> Tiziano Müller wrote:
> >> >> What is proposed in glep-55 seems to aim to solve both issues at the
> >> >> same time (it isn't stated) by switching file extension every time the
> >> >> eapi is changed. This is slightly against the principle of the least
> >> >> surprise and apparently is disliked by enough people to lead the
> >> >> situation to be discussed in the council.
> >> >>
> >> >
> >> > Instead of switching file extension every time the eapi is changed you
> >> > could also increment it only when a new EAPI breaks sourcing the ebuild
> >> > compared to the requirements of the prior EAPI.
> >> > (This way you'd in fact split EAPI into a major- and a minor-version.)
> >> >
> >>
> >> Doesn't that just add extra complexity for no gain.
> > Yes, sure. I was just looking for a solution for the "we have countless .eapi-X after 10 years" problem.
> 
> No one wants to be working with ebuild-29 or something like that in a
> few years and trying to figure out which feature came in which EAPI.
> Instead of bumping EAPI for each little change, save them up and bump
> no more than once a year or less, each bump bringing in some major new
> feature. With a little common sense and planning, we could make this a
> non-issue and give ebuild authors and PM devs alike a little time to
> get used to each change.

There also is the angle that deploying g55 requires waiting at least a 
full stage release (~year, at least by the old standards) to ensure 
people aren't screwed by the repository changing formats 
(unversioned!) under their feet.

I'd point out that g55 isn't even accepted (search the archives for 
the debates), but folks seem to be ignoring that and the issues of 
just flipping the switch...

~harring, aka the "version the farking repo format so stuff like this 
can be done cleanly" guy

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Tiziano Müller wrote:
>> What is proposed in glep-55 seems to aim to solve both issues at the 
>> same time (it isn't stated) by switching file extension every time the 
>> eapi is changed. This is slightly against the principle of the least 
>> surprise and apparently is disliked by enough people to lead the 
>> situation to be discussed in the council.
>>
> 
> Instead of switching file extension every time the eapi is changed you
> could also increment it only when a new EAPI breaks sourcing the ebuild
> compared to the requirements of the prior EAPI.
> (This way you'd in fact split EAPI into a major- and a minor-version.)

Makes you getting to have to do the two stage source again AND you get 
another non obvious condition "Should I bump the eapi internally or the 
filename?"

The main point again what is proposed in glep-55 is it that isn't 
invasive and non-transparent to users and developers.

As stated in the analysis, the user side is already covered by the fact 
users use the cache, the developer side would require a two stage 
sourcing when committing to remain transparent.

What we need to balance is if the invasive proposal is simpler than 
having a two stage sourcing done.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Thomas Anderson]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

On Mon, Feb 23, 2009 at 02:21:33PM +0100, Luca Barbato wrote:
> Tiziano M??ller wrote:
>>> What is proposed in glep-55 seems to aim to solve both issues at the same 
>>> time (it isn't stated) by switching file extension every time the eapi is 
>>> changed. This is slightly against the principle of the least surprise and 
>>> apparently is disliked by enough people to lead the situation to be 
>>> discussed in the council.
>>>
>> Instead of switching file extension every time the eapi is changed you
>> could also increment it only when a new EAPI breaks sourcing the ebuild
>> compared to the requirements of the prior EAPI.
>> (This way you'd in fact split EAPI into a major- and a minor-version.)
>
> Makes you getting to have to do the two stage source again AND you get 
> another non obvious condition "Should I bump the eapi internally or the 
> filename?"

The glep is quite clear on that point.

> The main point again what is proposed in glep-55 is it that isn't invasive 
> and non-transparent to users and developers.

It's not all that invasive. All that changes is that the EAPI goes at
the end of the filename and you don't set it in the ebuild. Developers
should be able to keep up with this, and if a user asks it's easy enough
to say that "it's a new version of ebuild, it has newer features see
www.blah.org/blah for details". And really, users already ask what EAPI
is so it's not that much headache.

-- 
---------
Thomas Anderson
Gentoo Developer
/////////
Areas of responsibility:
AMD64, Secretary to the Gentoo Council
---------

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Duncan]

Brian Harring <ferringb@gmail.com> posted 20090223085232.GA6492@hrair,
excerpted below, on  Mon, 23 Feb 2009 00:52:32 -0800:

> On Mon, Feb 23, 2009 at 09:38:06AM +0100, Tiziano M??ller wrote:
>> [quoting...]
>>> What is proposed in glep-55 seems to aim to solve both issues at the
>>> same time (it isn't stated) by switching file extension every time
>>> the eapi is changed. This is slightly against the principle of the
>>> least surprise and apparently is disliked by enough people[...]
>> 
>> Instead of switching file extension every time the eapi is changed you
>> could also increment it only when a new EAPI breaks sourcing the ebuild
>> compared to the requirements of the prior EAPI. (This way you'd in fact
>> split EAPI into a major- and a minor-version.)
> 
> Complicates the implementation (annoying), but more importantly negates
> one of the features of g55- being able to tell via the extension if the
> manager supports that EAPI.

That makes sense, but from my observation, the biggest resistance is 
coming from potentially unlimited changes to the extension.  That rubs 
some people strongly enough the wrong way to have folks threatening to 
leave over it, etc.  If it must be, it must be, but obviously, if there's 
a /reasonable/ way to avoid it, we should.

Way back when this first came up, I asked a simple question and IIRC 
wasn't satisfied with the answer.  Since the elements of it have been 
proposed separately, perhaps it's time to ask about the combination once 
again, as it seems to me to solve both the technical and aesthetic 
issues, tho admittedly it does have a bit of the "complicates the 
implementation" problem.

As I understand it, the nastiest technical problem is currently the 
chicken/egg issue of needing the EAPI to source the ebuild... to /get/ 
the EAPI.  Above, we have what amounts to a major/minor EAPI proposal, 
stick the major in the extension (or as a variant, the pre-extension 
filename), with it bumped only when the format changes enough to require 
pre-source knowledge of the change.

Elsewhere, someone proposed strenthening the format rules by hard-
specifying a location and format for the EAPI line, say line two of the 
ebuild and in a format specific enough to be parsed /without/ sourcing 
the ebuild, thus providing a pre-source method for grabbing the EAPI.  
The shoot-down when originally suggested was that this isn't flexible 
enough.  It's also arguably less efficient since one has to access the 
file twice, first to get the EAPI, then to actually source the file.  
Unfortunately the below suggestion doesn't avoid that.

Combining the two ideas, we get:

Why not put the "EAPI-major" aka "pre-parse-EAPI" in that hard-specified 
in-file location, thus giving the package manager a method to grab it pre-
source, then allow more flexibility on the "EAPI-minor" aka
"post-parse-EAPI"?

FWIW, all EAPIs to date have been EAPI-minor/post-parse, precisely 
/because/ of the need-the-EAPI-to-source-to-get-the-EAPI issue.  This 
would eliminate that issue, providing both the necessary pre-source 
(major) EAPI solution and flexibility on the post-source (minor) EAPI.  
It would also avoid the so controversial aesthetic issue, maintaining the 
traditional .ebuild extension.

The negative, however, as mentioned, is that of efficiency.  It'd be 
necessary to access the file twice, pre-source to get the EAPI-major, 
then the source to fill in all the details.  Putting just the EAPI-major 
in the filename/extension would avoid the first access (a dir listing 
would suffice) and using the filename for the EAPI entirely would in some 
cases possibly avoid accessing the file itself entirely -- at the expense 
of EAPI flexibility and aesthetics.


Meanwhile, a status/process observation, if I may.  Given the efficiency 
negative of putting the EAPI anywhere /but/ the filename and the way the 
debate has gone, GLEP-55 or something close to it (using the filename for 
EAPI) would appear headed toward ultimate approval, however slow it seems 
to be getting there.

The major blocker would appear to be that the GLEP as-is simply doesn't 
sufficiently address everything that has come up in the discussions.  As 
such, it's not clearly and sufficiently enough worded for the council to 
feel comfortable approving it.  Based on council logs and discussion, I 
get the STRONG impression that they are pretty much /begging/ the 
proponents to address this issue, updating the GLEP as appropriate, so 
they can /finally/ get out of the eternal debate stage and vote it up or 
down (presumably up as it doesn't appear they have a viable alternative 
either) on its merits, and the PMs can get it implemented and both the 
council and Gentoo can move on.

Unfortunately, due to "personnel issues", apparently there's currently 
nobody filling the triple qualifications of being (1) a strong enough 
proponent to bother, (2) a PM dev or other with sufficient grasp of the 
issues to actually /do/ the work, and (3) a Gentoo dev with the necessary 
authorization and access privs.  Until that changes and the GLEP is 
updated appropriately, we seem locked in the endless loop of discussion 
going nowhere, fast!

So it seems the proponents have a clear way forward, should they wish to 
pursue it.  Until they do, we might as well quit the discussion as it's 
not accomplishing anything, no matter how good it could be or how much of 
a block the failure to implement is on future improvements.  The council 
seems to have been clear enough, even /I'm/ getting that much.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

On Mon, 23 Feb 2009 04:26:49 -0800
Brian Harring <ferringb@gmail.com> wrote:
> There also is the angle that deploying g55 requires waiting at least
> a full stage release (~year, at least by the old standards) to ensure 
> people aren't screwed by the repository changing formats 
> (unversioned!) under their feet.

No it doesn't. It's transparent to users using an older package manager.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1892 bytes --]

On Mon, 23 Feb 2009 03:15:03 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> Let's try to start with a common workflow for the user:
> - an user with an ancient version of portage syncs
> - it requires a package
> - it looks at the cache ($portdir/metadata/cache/)
> - picks the best entry from the ones showing an eapi it understands
> - keeps going.
> 
> Apparently we do not have any issue...

...assuming the metadata cache is valid. That isn't always the case.

> 2- The user will get unpredictable behavior, but portage tell you
> when upgrading is needed...

Not if the version you'd need to do metadata generation is ~arch it
doesn't.

> 3- you'd have to disable them

Yes, tell everyone to disable all the overlays that make use of a few
features only in ~arch package managers... That'll work...

> In this case we have a problem if the source step is a single one, 
> portage won't know in advance how to behave.
> 
> So the first step has to be split in two:
> - first portage discovers which is the eapi version

...which it can't do, because it doesn't know the EAPI.

> The problem is that right now sourcing is done by having an
> instructed bash. So the simplest way to get the first step done is
> parsing the ebuild file with something different like file(1) and
> then instruct bash and do the parsing.

file(1) can't parse ebuilds. Only an ebuild implementation can parse
ebuilds, and only if it already knows the EAPI.

> What is proposed in glep-55 seems to aim to solve both issues at the 
> same time (it isn't stated) by switching file extension every time
> the eapi is changed. This is slightly against the principle of the
> least surprise and apparently is disliked by enough people to lead
> the situation to be discussed in the council.

There's no surprise at all. It's extremely clear.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

On Mon, Feb 23, 2009 at 01:50:10PM +0000, Ciaran McCreesh wrote:
> On Mon, 23 Feb 2009 04:26:49 -0800
> Brian Harring <ferringb@gmail.com> wrote:
> > There also is the angle that deploying g55 requires waiting at least
> > a full stage release (~year, at least by the old standards) to ensure 
> > people aren't screwed by the repository changing formats 
> > (unversioned!) under their feet.
> 
> No it doesn't. It's transparent to users using an older package manager.

Would be useful if someone pulled older portage versions and checked 
exactly what they do in this case- explode, behave, etc (manifest 
behaviour included).  It's been several years, but I recall portage 
having problems at the onset of EAPI w/ it.

Beyond that, what I was stating was that the user doesn't get told 
"sorry, your manager is too old, upgrade"- kind of an unobvious 
failing.

Frankly, in terms of g55 I don't particularly care if it were 
implemented- although I'd rather see it go in a seperate repo along w/ 
the dozen other fixups needed, preferably starting w/ overlays...

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

On Mon, 23 Feb 2009 06:15:25 -0800
Brian Harring <ferringb@gmail.com> wrote:
> > No it doesn't. It's transparent to users using an older package
> > manager.
> 
> Would be useful if someone pulled older portage versions and checked 
> exactly what they do in this case- explode, behave, etc (manifest 
> behaviour included).  It's been several years, but I recall portage 
> having problems at the onset of EAPI w/ it.

It was checked back when 55 was originally written. If it's broken now,
it's a breakage since then...

> Frankly, in terms of g55 I don't particularly care if it were 
> implemented- although I'd rather see it go in a seperate repo along
> w/ the dozen other fixups needed, preferably starting w/ overlays...

Well yes, but that's never realistically going to happen. 55's one of
the few repository format fixes that can happen in a reasonable
timeframe.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Richard Freeman]

Douglas Anderson wrote:
> No one wants to be working with ebuild-29 or something like that in a
> few years and trying to figure out which feature came in which EAPI.
> Instead of bumping EAPI for each little change, save them up and bump
> no more than once a year or less, each bump bringing in some major new
> feature. 

I got the impression that if anything there is a desire to allow EAPIs 
to change more offen, and not less.  And these changes could become more 
dramatic.  I'm not sure this is actually a bad thing (within reason - we 
do need to have clear specifications for anything that hits production 
so that we don't have a package manager mess).

Also - keep in mind that EAPIs do not need to be numbers and are not 
ordered.  You could have ebuild-i_am_a_furry_monkey or 
ebuild-<bunch-of-stuff-in-unicode-that-maybe-your-terminal-displays>. 
Sure - hopefully the names will be more sensible and somewhat uniform, 
but we're not necessarily just talking about adding a number to the 
extension.

I still don't see why we need to be encoding metadata in filenames. 
PERL doesn't care what a file extension is, python doesn't care, bzip2 
doesn't care, tar doesn't care, gzip doesn't care, and even ld-linux.so 
doesn't care.  I'm sure that in at least some of these cases they end up 
parsing parts of the file twice - once to figure out what it is, and the 
second time to actually handle it.  I'm actually hard pressed to think 
of any unix-based software that uses the filename to store a mandatory 
file format versioning specifier of some kind.

This seems to me to be a solved problem.  You put a header in a file 
that tells you how to read the file.  Headers are split into fields and 
if a program doesn't know what to do with a field it ignores it (or if 
the header so instructs it doesn't even try to parse the file).  This 
should be easy to do and keep the file bash-compatible.  Just stick a 
comment line close to the top of the file and put whatever you want on 
it.  You could also stick something in metadata.xml (although this makes 
working with ebuilds outside of a repository more difficult).  You run 
the file through an algorithm to find out what the EAPI is, and then 
source it if appropriate.

Sure, if you make some major change analogous to switching from the .rpm 
to the .deb package format then maybe an extension change would make 
sense.  But, why expose the inner workings of the package file format to 
the filesystem?

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2763 bytes --]

On Mon, 23 Feb 2009 09:28:06 -0500
Richard Freeman <rich0@gentoo.org> wrote:
> I got the impression that if anything there is a desire to allow
> EAPIs to change more offen, and not less.  And these changes could
> become more dramatic.

But we're still only talking maybe three new EAPIs a year.

> Also - keep in mind that EAPIs do not need to be numbers and are not 
> ordered.  You could have ebuild-i_am_a_furry_monkey or 
> ebuild-<bunch-of-stuff-in-unicode-that-maybe-your-terminal-displays>. 
> Sure - hopefully the names will be more sensible and somewhat
> uniform, but we're not necessarily just talking about adding a number
> to the extension.

You're using "developers might do something really stupid" as an
argument? By that logic, the current setup sucks since someone might
make an EAPI called a'b"c\d , so developers might have to put weird
escaping in when specifying EAPI.

Also note that kdebuild went with .kdebuild-1, not .ebuild-kdebuild-1.
It's no leap to have slightly different extension rules for any project
that creates its own non-numerical EAPIs.

> I still don't see why we need to be encoding metadata in filenames. 

Because the metadata has to be known before the file can be used.

> PERL doesn't care what a file extension is, python doesn't care,
> bzip2 doesn't care, tar doesn't care, gzip doesn't care, and even
> ld-linux.so doesn't care.  I'm sure that in at least some of these
> cases they end up parsing parts of the file twice - once to figure
> out what it is, and the second time to actually handle it.  I'm
> actually hard pressed to think of any unix-based software that uses
> the filename to store a mandatory file format versioning specifier of
> some kind.

Back when Perl 5 and PHP 5 were new, people often used extensions
like .php4 and .php5 to tell their webserver how to handle scripts...

> This seems to me to be a solved problem.  You put a header in a file 
> that tells you how to read the file.  Headers are split into fields
> and if a program doesn't know what to do with a field it ignores it
> (or if the header so instructs it doesn't even try to parse the
> file).  This should be easy to do and keep the file bash-compatible.
> Just stick a comment line close to the top of the file and put
> whatever you want on it.

That doesn't work with existing packages or existing package managers.

> Sure, if you make some major change analogous to switching from
> the .rpm to the .deb package format then maybe an extension change
> would make sense.  But, why expose the inner workings of the package
> file format to the filesystem?

For the same reason versions and package names are already in the
filename.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Mon, 23 Feb 2009 03:15:03 +0100
> Luca Barbato <lu_zero@gentoo.org> wrote:
>> Let's try to start with a common workflow for the user:
>> - an user with an ancient version of portage syncs
>> - it requires a package
>> - it looks at the cache ($portdir/metadata/cache/)
>> - picks the best entry from the ones showing an eapi it understands
>> - keeps going.
>>
>> Apparently we do not have any issue...
> 
> ...assuming the metadata cache is valid. That isn't always the case.

When it isn't?

>> 2- The user will get unpredictable behavior, but portage tell you
>> when upgrading is needed...
> 
> Not if the version you'd need to do metadata generation is ~arch it
> doesn't.

...

>> 3- you'd have to disable them
> 
> Yes, tell everyone to disable all the overlays that make use of a few
> features only in ~arch package managers... That'll work...

disable overlays to UPGRADE to the new portage. Not rocket science...

>> In this case we have a problem if the source step is a single one, 
>> portage won't know in advance how to behave.
>>
>> So the first step has to be split in two:
>> - first portage discovers which is the eapi version
> 
> ...which it can't do, because it doesn't know the EAPI.

If you are generating the cache you must have a way to know it...

>> The problem is that right now sourcing is done by having an
>> instructed bash. So the simplest way to get the first step done is
>> parsing the ebuild file with something different like file(1) and
>> then instruct bash and do the parsing.
> 
> file(1) can't parse ebuilds.

file parses quite well avi and mov, ebuild will be anytime more complex 
than that right?

Anyway it isn't a problem since the version of portage doing the work is 
supposed to be up to date, if isn't it will be updated first using the 
normal user workflow that has already been covered by the cache.

> Only an ebuild implementation can parse
> ebuilds, and only if it already knows the EAPI.

That is always the case since you are adding an ebuild and you are 
supposed to have an up to date portage in order to do that.

>> What is proposed in glep-55 seems to aim to solve both issues at the 
>> same time (it isn't stated) by switching file extension every time
>> the eapi is changed. This is slightly against the principle of the
>> least surprise and apparently is disliked by enough people to lead
>> the situation to be discussed in the council.
> 
> There's no surprise at all. It's extremely clear.

Not that much.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

[gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3470 bytes --]

On Mon, 23 Feb 2009 15:46:24 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> >> Apparently we do not have any issue...
> > 
> > ...assuming the metadata cache is valid. That isn't always the case.
> 
> When it isn't?

Every now and again (probably after someone changes eutils...), rsync
mirrors end up shipping a load of stale metadata for parts of the tree.
Portage users probably don't notice, since it just causes a slowdown.

> disable overlays to UPGRADE to the new portage. Not rocket science...

No no. You're forcing people to switch to ~arch to be able to use
overlays.

> >> In this case we have a problem if the source step is a single one, 
> >> portage won't know in advance how to behave.
> >>
> >> So the first step has to be split in two:
> >> - first portage discovers which is the eapi version
> > 
> > ...which it can't do, because it doesn't know the EAPI.
> 
> If you are generating the cache you must have a way to know it...

No, we don't. That's the entire frickin' point of GLEP 55, and one
that you would do well to understand fully before commenting further.
The way we generate metadata now is massively horrible, and only works
because there aren't any serious global scope differences between
EAPIs. We can't make any global scope changes to future EAPIs because
it breaks current metadata generation. Right now it's safe to pretend
EAPI 0 until the real EAPI is worked out, but that won't hold in the
future -- as soon as global scope changes come along, there's no safe
EAPI that can be assumed, even if we didn't have to worry about
breaking existing package managers.

> >> The problem is that right now sourcing is done by having an
> >> instructed bash. So the simplest way to get the first step done is
> >> parsing the ebuild file with something different like file(1) and
> >> then instruct bash and do the parsing.
> > 
> > file(1) can't parse ebuilds.
> 
> file parses quite well avi and mov, ebuild will be anytime more
> complex than that right?

It's already *way* more complicated than that. Extracting metadata
requires a full bash 3 implementation along with a considerable amount
of supporting code.

> Anyway it isn't a problem since the version of portage doing the work
> is supposed to be up to date, if isn't it will be updated first using
> the normal user workflow that has already been covered by the cache.

Most users don't run ~arch, and do use overlays. So no, this will affect
normal user workflow.

> > Only an ebuild implementation can parse
> > ebuilds, and only if it already knows the EAPI.
> 
> That is always the case since you are adding an ebuild and you are 
> supposed to have an up to date portage in order to do that.

Again, you're not getting it. Doesn't matter how up to date your
package manager is. You can't find out the EAPI unless you already know
the EAPI.

> >> What is proposed in glep-55 seems to aim to solve both issues at
> >> the same time (it isn't stated) by switching file extension every
> >> time the eapi is changed. This is slightly against the principle
> >> of the least surprise and apparently is disliked by enough people
> >> to lead the situation to be discussed in the council.
> > 
> > There's no surprise at all. It's extremely clear.
> 
> Not that much.

How is '.ebuild-3' being used to specify version 3 of the ebuild format
in the least bit surprising?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Steve Dibb]

Richard Freeman wrote:
> I still don't see why we need to be encoding metadata in filenames. PERL 
> doesn't care what a file extension is, python doesn't care, bzip2 
> doesn't care, tar doesn't care, gzip doesn't care, and even ld-linux.so 
> doesn't care.  I'm sure that in at least some of these cases they end up 
> parsing parts of the file twice - once to figure out what it is, and the 
> second time to actually handle it.  I'm actually hard pressed to think 
> of any unix-based software that uses the filename to store a mandatory 
> file format versioning specifier of some kind.

I have to admit I'm in the same camp with Richard, and don't understand 
the necessity.  I'm also opposed to creating arbitrary suffixes to the 
ebuild extension, for cosmetic and compatibility reasons.

Plus, I don't really grasp the whole "we have to source the whole ebuild 
to know the EAPI version" argument.  It's one variable, in one line. 
Can't a simple parser get that and go from there?

Steve

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On Mon, 23 Feb 2009 08:43:09 -0700
Steve Dibb <beandog@gentoo.org> wrote:
> Plus, I don't really grasp the whole "we have to source the whole
> ebuild to know the EAPI version" argument.  It's one variable, in one
> line. Can't a simple parser get that and go from there?

Not true. This is entirely legal:

In pkg-1.ebuild:

    EAPI="${PV}"
    printf -v EAPI '%s' 4
    inherit foo
    EAPI="2"

In foo.eclass:

    EAPI="3"

And in this case, the package manager has to know that EAPI=2, and it
has to use EAPI 2's behaviour for performing the inherit.

In fact, it gets worse if you consider that future EAPIs will probably
allow per-package eclasses. So you could end up with this crazy
situation:

In pkg-1.ebuild:

    require pkg

In cat/pkg/pkg.eclass:

    EAPI="3"

In global pkg.eclass:

    EAPI="2"

So here you can't even use a faked pre-source EAPI. If you assume EAPI
0 beforehand, the global pkg.eclass will be used, so EAPI will end up
as 2. But if you assume EAPI 3 beforehand, the per-pkg eclass will be
used, so the EAPI will end up as 3.

It gets even crazier if you put EAPI="2" in the per-pkg eclass and
EAPI="3" in the global one instead...

And whilst this is clearly a deliberate example of how to create
craziness, the only difference between this and the real world is that
the craziness is obvious here. The current rules really are this
complicated, and we can't retroactively fix them.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Peter Alfredsen]

On Mon, 23 Feb 2009 08:43:09 -0700
Steve Dibb <beandog@gentoo.org> wrote:

> Plus, I don't really grasp the whole "we have to source the whole
> ebuild to know the EAPI version" argument.  It's one variable, in one
> line. Can't a simple parser get that and go from there?

The problem is that its technically allowed for the value of $EAPI to be
dependent on other bits and pieces. For instance:

if [[ $PV = 2.6 ]]
then
	EAPI=2
else
	EAPI=1
fi

The quick-n-dirty solution to that is to just say that EAPI is not just
a bash variable, it's also a magic string, so manipulating it in bash
is forbidden. And please place it in line 5 of your ebuild, kthxbye.

To be honest I see no good reason for allowing manipulation of it, but
I'm sure other people will tell me why adding this requirement at this
point is wrong even though no ebuilds in the tree to the best of my
knowledge use EAPI as anything more than a declaration that's placed
Just before inherit, right after the header.

/loki_val

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

On Mon, 23 Feb 2009 17:06:17 +0100
Peter Alfredsen <loki_val@gentoo.org> wrote:
> To be honest I see no good reason for allowing manipulation of it, but
> I'm sure other people will tell me why adding this requirement at this
> point is wrong

There's not really a good reason to allow manipulating it (and,
obviously, with GLEP 55 manipulating it becomes impossible), but since
for all current EAPIs it's just defined as a metadata variable that's
generated in the same way as things like SLOT, manipulating it is
unfortunately legal.

> even though no ebuilds in the tree to the best of my knowledge use
> EAPI as anything more than a declaration that's placed Just before
> inherit, right after the header.

People have, in the past, set EAPI inside eclasses. It's stupid and
horrible, but they've done it.

But here's the thing -- even if we retroactively enforce a new rule
requiring it to be specified in a particular way right after the header
(which is bad, since it breaks things people have already done), it
*still* doesn't let us change global scope behaviour since current
package managers don't extract EAPI the horrid way.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 1233 bytes --]

On Mon, 23 Feb 2009 15:53:20 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Mon, 23 Feb 2009 08:43:09 -0700
> Steve Dibb <beandog@gentoo.org> wrote:
> > Plus, I don't really grasp the whole "we have to source the whole
> > ebuild to know the EAPI version" argument.  It's one variable, in
> > one line. Can't a simple parser get that and go from there?
> 
> Not true. This is entirely legal:
> 
> In pkg-1.ebuild:
> 
>     EAPI="${PV}"
>     printf -v EAPI '%s' 4
>     inherit foo
>     EAPI="2"

Which begs the question: is it really worth allowing it?
If we only allow constant assignments (which is an implicit restriction
in the file extension version) then this can be parsed easily with
grep/tr/awk/etc and can be the magic eapi guessing. Of course the tree
has to be checked before implementing this and we'll have to wait a
good amount of time before breaking the current eapi bash-parsing but
I'm not aware of any eapi proposal that would break the current behavior
and would be usable in the main tree within a reasonable amount of
time such that we can't ignore backward compatibility.


> In foo.eclass:
> 
>     EAPI="3"

I thought this was prohibited.

Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

On Mon, 23 Feb 2009 17:13:16 +0100
Alexis Ballier <aballier@gentoo.org> wrote:
> Which begs the question: is it really worth allowing it?
> If we only allow constant assignments (which is an implicit
> restriction in the file extension version) then this can be parsed
> easily with grep/tr/awk/etc and can be the magic eapi guessing. Of
> course the tree has to be checked before implementing this and we'll
> have to wait a good amount of time before breaking the current eapi
> bash-parsing but I'm not aware of any eapi proposal that would break
> the current behavior and would be usable in the main tree within a
> reasonable amount of time such that we can't ignore backward
> compatibility.

...and then we have to do the whole thing again every time something
new crops up. EAPI was supposed to solve this, and profile eapi and GLEP
55 finish the job. Repeatedly going back and saying "oh, we have to
wait another year or more again" is unacceptable.

> > In foo.eclass:
> > 
> >     EAPI="3"
> 
> I thought this was prohibited.

It's legal, and people have done it, but it's considered by most people
to be a horrible QA violation.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

On Mon, 23 Feb 2009 16:19:56 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Mon, 23 Feb 2009 17:13:16 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
> > Which begs the question: is it really worth allowing it?
> > If we only allow constant assignments (which is an implicit
> > restriction in the file extension version) then this can be parsed
> > easily with grep/tr/awk/etc and can be the magic eapi guessing. Of
> > course the tree has to be checked before implementing this and we'll
> > have to wait a good amount of time before breaking the current eapi
> > bash-parsing but I'm not aware of any eapi proposal that would break
> > the current behavior and would be usable in the main tree within a
> > reasonable amount of time such that we can't ignore backward
> > compatibility.
> 
> ...and then we have to do the whole thing again every time something
> new crops up.

Please give an example because I fail to see how.

> EAPI was supposed to solve this, and profile eapi and
> GLEP 55 finish the job. Repeatedly going back and saying "oh, we have
> to wait another year or more again" is unacceptable.

Had we found a compromise at the beginning of glep55, that extra year
would be over by now...


Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 943 bytes --]

On Mon, 23 Feb 2009 17:48:27 +0100
Alexis Ballier <aballier@gentoo.org> wrote:
> > ...and then we have to do the whole thing again every time something
> > new crops up.
> 
> Please give an example because I fail to see how.

New version suffix rules. New bash versions. New package naming rules.
Partially composable EAPIs. Tree-provided internals. Consistent variable
namespacing. Metadata via function calls.

> > EAPI was supposed to solve this, and profile eapi and
> > GLEP 55 finish the job. Repeatedly going back and saying "oh, we
> > have to wait another year or more again" is unacceptable.
> 
> Had we found a compromise at the beginning of glep55, that extra year
> would be over by now...

And we'd be starting on the next batch of "oh, we need to wait another
year". Had GLEP 55's necessity been accepted a year ago, we'd have a
whole bunch of requested features implemented by now.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alec Warner]

On Mon, Feb 23, 2009 at 7:53 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> On Mon, 23 Feb 2009 08:43:09 -0700
> Steve Dibb <beandog@gentoo.org> wrote:
>> Plus, I don't really grasp the whole "we have to source the whole
>> ebuild to know the EAPI version" argument.  It's one variable, in one
>> line. Can't a simple parser get that and go from there?
>
> Not true. This is entirely legal:
>
> In pkg-1.ebuild:
>
>    EAPI="${PV}"
>    printf -v EAPI '%s' 4
>    inherit foo
>    EAPI="2"
>
> In foo.eclass:
>
>    EAPI="3"

This is not legal, the devmanual[1] explicitly states that it is not
legal to set EAPI in an eclass.  If it was you would have an open and
shut case for g55 (since you could set EAPI based on PV in an eclass
and then inherit the eapi; requiring a bash parser).  But its illegal,
which makes your argument harder.

A bad argument is EAPI based on some PV logic that you can copy across
ebuilds that differ by version; which would fall into 'stupid' in my
book.

[1] http://devmanual.gentoo.org/ebuild-writing/eapi/index.html

>
> And in this case, the package manager has to know that EAPI=2, and it
> has to use EAPI 2's behaviour for performing the inherit.
>
> In fact, it gets worse if you consider that future EAPIs will probably
> allow per-package eclasses. So you could end up with this crazy
> situation:
>
> In pkg-1.ebuild:
>
>    require pkg
>
> In cat/pkg/pkg.eclass:
>
>    EAPI="3"
>
> In global pkg.eclass:
>
>    EAPI="2"

Also illegal, but I assume here you intend to change the legality of
things in your weird future example.

>
> So here you can't even use a faked pre-source EAPI. If you assume EAPI
> 0 beforehand, the global pkg.eclass will be used, so EAPI will end up
> as 2. But if you assume EAPI 3 beforehand, the per-pkg eclass will be
> used, so the EAPI will end up as 3.
>
> It gets even crazier if you put EAPI="2" in the per-pkg eclass and
> EAPI="3" in the global one instead...
>
> And whilst this is clearly a deliberate example of how to create
> craziness, the only difference between this and the real world is that
> the craziness is obvious here. The current rules really are this
> complicated, and we can't retroactively fix them.
>
> --
> Ciaran McCreesh
>

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 468 bytes --]

On Mon, 23 Feb 2009 11:02:17 -0800
Alec Warner <antarus@gentoo.org> wrote:
> > In foo.eclass:
> >
> >    EAPI="3"
> 
> This is not legal, the devmanual[1] explicitly states that it is not
> legal to set EAPI in an eclass.

That's purely a QA thing, and it only applies to repositories that
follow Gentoo's QA rules. It's in the same category as rules about
what indenting style to use, not rules about how variables are
formatted.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 2079 bytes --]

On Mon, 23 Feb 2009 08:43:09 -0700
Steve Dibb <beandog@gentoo.org> wrote:

> Richard Freeman wrote:
> > I still don't see why we need to be encoding metadata in filenames.
> > PERL doesn't care what a file extension is, python doesn't care,
> > bzip2 doesn't care, tar doesn't care, gzip doesn't care, and even
> > ld-linux.so doesn't care.  I'm sure that in at least some of these
> > cases they end up parsing parts of the file twice - once to figure
> > out what it is, and the second time to actually handle it.  I'm
> > actually hard pressed to think of any unix-based software that uses
> > the filename to store a mandatory file format versioning specifier
> > of some kind.

$ ls /usr/lib

> I have to admit I'm in the same camp with Richard, and don't
> understand the necessity.  I'm also opposed to creating arbitrary
> suffixes to the ebuild extension, for cosmetic and compatibility
> reasons.
> 
> Plus, I don't really grasp the whole "we have to source the whole
> ebuild to know the EAPI version" argument.  It's one variable, in one
> line. Can't a simple parser get that and go from there?

Not really.  Let's play Guess the EAPI. :o

1.
-----
EAPI=1
----

2. (with myeclass.eclass containing EAPI=2)
-----
EAPI=1
inherit myeclass
-----

3. (with myeclass.eclass containing EAPI=2)
-----
EAPI=5
inherit myeclass
-----


1.  1  (simple enough)
2.  2  (because myeclass.eclass sets EAPI=2)
3.  5  (inherit was changed in EAPI 5 to not overwrite ${EAPI} if
        already set in an ebuild)

So you see, it's not as easy as a grep command.  You need to source the
ebuild to know how things like inherit will affect the environment.
And without knowing the EAPI, you don't know which version of inherit to
call.

(i hope i have this right.  feel free to call me names if i don't)


-- 
gcc-porting,                                      by design, by neglect
treecleaner,                              for a fact or just for effect
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

Ciaran McCreesh wrote:
> On Mon, 23 Feb 2009 17:48:27 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
>>> ...and then we have to do the whole thing again every time something
>>> new crops up.
>> Please give an example because I fail to see how.
> 
> New version suffix rules. New bash versions. New package naming rules.
> Partially composable EAPIs. Tree-provided internals. Consistent variable
> namespacing. Metadata via function calls.
> 
>>> EAPI was supposed to solve this, and profile eapi and
>>> GLEP 55 finish the job. Repeatedly going back and saying "oh, we
>>> have to wait another year or more again" is unacceptable.
>> Had we found a compromise at the beginning of glep55, that extra year
>> would be over by now...
> 
> And we'd be starting on the next batch of "oh, we need to wait another
> year". Had GLEP 55's necessity been accepted a year ago, we'd have a
> whole bunch of requested features implemented by now.
> 

I doubt Portage would have gained new features any faster.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

On Mon, 23 Feb 2009 21:30:04 +0200
Petteri Räty <betelgeuse@gentoo.org> wrote:
> > And we'd be starting on the next batch of "oh, we need to wait
> > another year". Had GLEP 55's necessity been accepted a year ago,
> > we'd have a whole bunch of requested features implemented by now.
> > 
> 
> I doubt Portage would have gained new features any faster.

Some useful things that are currently difficult become a lot easier to
implement with GLEP 55. Per-package eclasses, for example, are easy if
you don't have to care about the upgrade path, as is replacing
versionator with a package manager internal. The complexity for both of
those is in the upgrade path, not the implementation.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ryan Hill wrote:
> On Mon, 23 Feb 2009 08:43:09 -0700
> Steve Dibb <beandog@gentoo.org> wrote:
> 
>> Richard Freeman wrote:
>>> I still don't see why we need to be encoding metadata in filenames.
>>> PERL doesn't care what a file extension is, python doesn't care,
>>> bzip2 doesn't care, tar doesn't care, gzip doesn't care, and even
>>> ld-linux.so doesn't care.  I'm sure that in at least some of these
>>> cases they end up parsing parts of the file twice - once to figure
>>> out what it is, and the second time to actually handle it.  I'm
>>> actually hard pressed to think of any unix-based software that uses
>>> the filename to store a mandatory file format versioning specifier
>>> of some kind.
> 
> $ ls /usr/lib

ldconfig ?

>> I have to admit I'm in the same camp with Richard, and don't
>> understand the necessity.  I'm also opposed to creating arbitrary
>> suffixes to the ebuild extension, for cosmetic and compatibility
>> reasons.
>>
>> Plus, I don't really grasp the whole "we have to source the whole
>> ebuild to know the EAPI version" argument.  It's one variable, in one
>> line. Can't a simple parser get that and go from there?
> 
> Not really.  Let's play Guess the EAPI. :o
> 
> 1.
> -----
> EAPI=1
> ----
> 
> 2. (with myeclass.eclass containing EAPI=2)
> -----
> EAPI=1
> inherit myeclass

Invalid

> -----
> 
> 3. (with myeclass.eclass containing EAPI=2)
> -----
> EAPI=5
> inherit myeclass

Invalid

> So you see, it's not as easy as a grep command.  You need to source the
> ebuild to know how things like inherit will affect the environment.
> And without knowing the EAPI, you don't know which version of inherit to
> call.

It it isn't invalid already...

> 
> (i hope i have this right.  feel free to call me names if i don't)

Names!

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

On Mon, 23 Feb 2009 21:51:11 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> > 2. (with myeclass.eclass containing EAPI=2)
> > -----
> > EAPI=1
> > inherit myeclass
> 
> Invalid

QA violation, but legal and a pain in the ass.

> > 3. (with myeclass.eclass containing EAPI=2)
> > -----
> > EAPI=5
> > inherit myeclass
> 
> Invalid

QA violation, but legal and a pain in the ass.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]

On Mon, 23 Feb 2009 20:54:38 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Mon, 23 Feb 2009 21:51:11 +0100
> Luca Barbato <lu_zero@gentoo.org> wrote:
> > > 2. (with myeclass.eclass containing EAPI=2)
> > > -----
> > > EAPI=1
> > > inherit myeclass
> > 
> > Invalid
> 
> QA violation, but legal and a pain in the ass.

I didn't think it was a brainy thing to do, but I can't find anything
saying it isn't allowed.  It probably shouldn't be.

> > > 3. (with myeclass.eclass containing EAPI=2)
> > > -----
> > > EAPI=5
> > > inherit myeclass
> > 
> > Invalid
> 
> QA violation, but legal and a pain in the ass.
> 

Can we ban eclasses from setting EAPI?  Is there any case where it
would be sane?


-- 
gcc-porting,                                      by design, by neglect
treecleaner,                              for a fact or just for effect
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]

On Mon, 23 Feb 2009 16:15:25 -0600
Ryan Hill <dirtyepic@gentoo.org> wrote:

> On Mon, 23 Feb 2009 20:54:38 +0000
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> 
> > On Mon, 23 Feb 2009 21:51:11 +0100
> > Luca Barbato <lu_zero@gentoo.org> wrote:
> > > > 2. (with myeclass.eclass containing EAPI=2)
> > > > -----
> > > > EAPI=1
> > > > inherit myeclass
> > > 
> > > Invalid
> > 
> > QA violation, but legal and a pain in the ass.
> 
> I didn't think it was a brainy thing to do, but I can't find anything
> saying it isn't allowed.  It probably shouldn't be.

Ah now i did.  

-- 
gcc-porting,                                      by design, by neglect
treecleaner,                              for a fact or just for effect
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]

On Mon, 23 Feb 2009 16:15:25 -0600
Ryan Hill <dirtyepic@gentoo.org> wrote:
> Can we ban eclasses from setting EAPI?  Is there any case where it
> would be sane?

It's already banned from a QA perspective, but from a package manager
perspective people have done it in the past and possibly still do do
it, and the spec doesn't forbid it.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Richard Freeman]

Ryan Hill wrote:
>> Richard Freeman wrote:
>>> I'm
>>> actually hard pressed to think of any unix-based software that uses
>>> the filename to store a mandatory file format versioning specifier
>>> of some kind.
> 
> $ ls /usr/lib


I was referring to a file FORMAT versioning scheme - not a file CONTENT 
versioning scheme.  The formats of all the files in /usr/lib are 
generally identical.  Where they vary it has nothing to do with their 
filenames.  The reason for the version in the filenames is that the 
content is versioned.

The dynamic linker doesn't need to consult the filename to figure out 
how to parse a shared object.  It only consults the filename to figure 
out which shared object is needed.  That is actually analogous to 
putting the package name/version in the ebuild filename.

In any case, I'm not trying to say that these issues absolutely prevent 
the adoption of GLEP 55.  It just leaves a sour taste in my mouth, and 
keeps nagging at me that there must be a better way.

I'd rather see the number of filename variations be kept to a minimum. 
Sure, if we were talking about a one-time change from ebuild to ebuild2 
and in five years a change to ebuild3 then that would be one thing.  It 
seems like we could be up to ebuild-kde4-3.2 in six months.

And I don't mean to suggest that I don't think that efforts would be 
made to keep things sensible.  It just seems like once we start down 
that road it will be hard to turn around if we don't like where we end up.

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 283 bytes --]

On Mon, 23 Feb 2009 18:47:07 -0500
Richard Freeman <rich0@gentoo.org> wrote:
> It seems like we could be up to ebuild-kde4-3.2 in six months.

Why on earth do people think that? Of all the crazy being thrown
around, this is the only one wearing a tutu.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Richard Freeman]

Ciaran McCreesh wrote:
> On Mon, 23 Feb 2009 18:47:07 -0500
> Richard Freeman <rich0@gentoo.org> wrote:
>> It seems like we could be up to ebuild-kde4-3.2 in six months.
> 
> Why on earth do people think that? Of all the crazy being thrown
> around, this is the only one wearing a tutu.
> 

I suppose I'm exaggerating a little deliberately to make a point.  It 
isn't so much that I don't think that people designing the extensions 
won't use sense, but that we're still potentially facing multiple new 
file extensions per year.  Maybe not 15, but certainly 1-3.  That can 
add up fast.  If we had been doing this all along then we'd probably 
expect there to be upwards of 10-20 file extensions in portage today.

It just seems like it isn't the best solution.  You can get the same 
effect by just sticking something in a comment line a few lines into the 
ebuild in a fixed position.  Sure, the file might need to be read twice, 
but unless the reading takes place widely separated in time the file is 
going to be in the cache the second time around.  With proper caching 
you only need to scan files that have changed - we can't have that many 
daily commits, can we?

I'll probably refrain from commenting further - I trust the council to 
weigh all the options and go with whatever makes the most sense. 
However, I did want to make it clear that I don't think that the folks 
advocating this approach are out to release 47 EAPI releases per year or 
anything...

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

On Mon, 23 Feb 2009 19:17:19 -0500
Richard Freeman <rich0@gentoo.org> wrote:
> It just seems like it isn't the best solution.  You can get the same 
> effect by just sticking something in a comment line a few lines into
> the ebuild in a fixed position.

No you can't. It doesn't work with existing package managers, and it
doesn't let you change name or version rules.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Jim Ramsay]

Alistair Bush <ali_bush@gentoo.org> wrote:
> Tiziano Müller wrote:
> > Instead of switching file extension every time the eapi is changed
> > you could also increment it only when a new EAPI breaks sourcing
> > the ebuild compared to the requirements of the prior EAPI.
> > (This way you'd in fact split EAPI into a major- and a
> > minor-version.)
> 
> Doesn't that just add extra complexity for no gain.

Actually, I think there would be a huge gain.

The gain would be that suddenly all those who oppose glep-55 because
they're afraid the filename suffix will change too often will suddenly
have nothing to worry about.

For those who think glep-55 is the right thing to do, it really
*is* glep-55, but with a small caveat that we shouldn't just change the
filename extension for every single little feature enhancement.

-- 
Jim Ramsay
Gentoo/Linux Developer (rox,gkrellm,fluxbox,vim)

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Richard Freeman]

Ciaran McCreesh wrote:
> On Mon, 23 Feb 2009 19:17:19 -0500
> Richard Freeman <rich0@gentoo.org> wrote:
>> It just seems like it isn't the best solution.  You can get the same 
>> effect by just sticking something in a comment line a few lines into
>> the ebuild in a fixed position.
> 
> No you can't. It doesn't work with existing package managers,

Agreed.  This would require some delay in implementation and would 
require users to have some minimal package manager version to handle 
major changes in a repository.

 > and it
> doesn't let you change name or version rules.
> 

Neither does putting the EAPI in the filename as far as I can tell.  It 
isn't like you want to have ebuild filenames like:

foo-1.1.ebuild-\{EAPI\=1\ \;\ if\ \[\[\ \$PV\ =\ 2.6\ \]\]\ \;\ then\ 
EAPI\=2\ \;\ fi\}

Putting the EAPI in the filename forces it to be a rigid constant for 
the purposes of determining how to parse the file.  Putting the EAPI in 
a comment line does the same.  Both allow for dynamic manipulation of 
the variable at a later stage of parsing, but this is after the package 
manager has committed to sourcing the file in some particular manner. 
If anything you get more flexibility putting it inside the file since at 
least you can make it very long without clogging up command lines.

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Luca Barbato wrote:
> Ciaran McCreesh wrote:
>> Because your proposal addresses none of the underlying problems which
>> GLEP 55 was created to solve.

let's get some numbers to have an idea of the dimension of the problem.

domino portage # wc -l /dev/shm/eapi_files.list
2854 /dev/shm/eapi_files.list

domino portage # ls *-*/*/*.ebuild | wc -l
25761

domino portage # grep -l EAPI eclass/*.eclass | wc -l
22

domino portage # ls eclass/*.eclass | wc -l
240

there aren't eclasses setting EAPI directly.

eapi is set either using EAPI=X or EAPI="X"

domino portage # time grep EAPI *-*/*/*.ebuild > /dev/shm/eapi_files.list

real	0m1.019s
user	0m0.608s
sys	0m0.412s

domino portage # time (for a in *-*/*/*.ebuild*; do echo ${A##*.ebuild}; 
done) > /dev/null

real	0m0.916s
user	0m0.764s
sys	0m0.152s

domino portage # time emerge --regen > /dev/shm/regen

real	0m9.308s
user	0m7.648s
sys	0m1.664s


Restricting eapi so it could surely parsed using something as
complex as grep would and using a two stage parsing would increase to 
about 1/9

Using a dumb way to extract the eapi from extension seems to take 1/10

Is there any technical merit in putting eapi in the file extension while 
we could restrict the format the same way in file and have about the 
same, negligible, performance hit? (I used warm cache since you need the 
file anyway so you don't spend time to look it up twice or put it in 
cache twice)

Please come up with other numbers or saner implementations to compare.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alistair Bush]

Luca Barbato wrote:
> Luca Barbato wrote:
>> Ciaran McCreesh wrote:
>>> Because your proposal addresses none of the underlying problems which
>>> GLEP 55 was created to solve.
> 
> let's get some numbers to have an idea of the dimension of the problem.
> 

I just don't think those numbers tell us anything and that should be
obvious from anyone who has read GLEP 55[1],  we ain't really attempting
to solve a problem that exists within the tree currently (well the bash
issue does, in a way ). We are trying to solve issues that ware stopping
the "tree" moving forward.  Lets evaluate GLEP 55 in the problems it is
attempting to solve.

[1]
Problem

The current way of specifying the EAPI in ebuilds is flawed. In order to
get the EAPI the package manager needs to source the ebuild, which
itself needs the EAPI in the first place. Otherwise it imposes a serious
limitation, namely every ebuild, using any of the future EAPIs, will
have to be source'able by old package managers and hence there is no way
to do any of the following:

        * Change the behaviour of inherit in any way (for example, to
extend or change eclass functionality).
        * Add new global scope functions in any sane way.
        * Extend versioning rules in an EAPI - for example, addition of
the scm suffix - GLEP54 [1].

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Alistair Bush wrote:
> I just don't think those numbers tell us anything and that should be
> obvious from anyone who has read GLEP 55[1],  we ain't really attempting
> to solve a problem that exists within the tree currently (well the bash
> issue does, in a way ). We are trying to solve issues that ware stopping
> the "tree" moving forward.  Lets evaluate GLEP 55 in the problems it is
> attempting to solve.

I'm afraid you missed the whole point...

- what is in the proposal is a solution looking for a problem: nobody 
updated the glep with the required sections, nobody put up a list of 
bugs/rfe from bugzilla it helps to solve. Vague "leading to the future 
change" declaration aren't what I'd expect.

- Assuming there is an actual reason to move forward (by digging 
bugzilla yourself or deciding to do so as academic exercise) you could 
think about the problem and its solutions (my the email starting this 
thread on dev)

- Given all you need is just to have a way to get the information about 
EAPI before you actually parse the ebuild since the eapi defines how you 
parse it, you can come up with various solutions, the simplest being 
first extract the eapi, being it in a fixed place, and then do the parse.

- Extracting such information could have different costs depending on 
where to place it.

- I started to check if the proposal about having the fixed position as 
the end of the filename is really much more viable than having it at the 
top of the file.

lu


-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[George Shapovalov]

(Ok this thread grew too long, so I gotta chime in :))

We could start using extended attributes or mandate reiser4 for portage dir or 
some other special "in between" (the inside of file and its name) feature..

Sorry for the noise and insane "implementation" suggestion :)..

George

PS
Actually, come to think of it, reizer4 might be just the right tool for the 
job (for keeping the volatile info split into bunch of small entries), 
hypothetically of course..


Tuesday, 24. February 2009, Luca Barbato Ви написали:
> Luca Barbato wrote:
> > Ciaran McCreesh wrote:
> >> Because your proposal addresses none of the underlying problems which
> >> GLEP 55 was created to solve.
>
> let's get some numbers to have an idea of the dimension of the problem.
[skip]
>
> Please come up with other numbers or saner implementations to compare.
>
> lu

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alistair Bush]

George Shapovalov wrote:
> (Ok this thread grew too long, so I gotta chime in :))
> 
> We could start using extended attributes or mandate reiser4 for portage dir or 
> some other special "in between" (the inside of file and its name) feature..

No.
1) I wouldn't use reiser4 so that would be the end of that.
2) how well do rsync and cvs support xattr's.  How about linux support
verses bsd, or windows even.
3) It is just a bad solution

> 
> Sorry for the noise and insane "implementation" suggestion :)..

At least you realise it :)

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alistair Bush]

Luca Barbato wrote:
> Alistair Bush wrote:
>> I just don't think those numbers tell us anything and that should be
>> obvious from anyone who has read GLEP 55[1],  we ain't really attempting
>> to solve a problem that exists within the tree currently (well the bash
>> issue does, in a way ). We are trying to solve issues that ware stopping
>> the "tree" moving forward.  Lets evaluate GLEP 55 in the problems it is
>> attempting to solve.
> 
> I'm afraid you missed the whole point...
> 
> - what is in the proposal is a solution looking for a problem: nobody
> updated the glep with the required sections, nobody put up a list of
> bugs/rfe from bugzilla it helps to solve. Vague "leading to the future
> change" declaration aren't what I'd expect.
> 

So im mean't to start committing ebuilds into the tree that expect a
certain unimplemented functionality, only to file bugs against portage
for it not supporting them?  Or can we be smart enough to realise that
there are limitation to the current standard and then attempt to fix
them before they become a problem.  Plus we already know of at least one
case where we will encounter a problem in the future,  why?  because we
have already.  Not sure if there is a open bug about it tho.

This actually eats at me,  your basically saying GLEP's should only be
reactive.  Why don't we all just roll over and die.

> - Assuming there is an actual reason to move forward (by digging
> bugzilla yourself or deciding to do so as academic exercise) you could
> think about the problem and its solutions (my the email starting this
> thread on dev)

I have already considered the problems, and believe GLEP 55 is the
**best** solution to them.  Is it perfect, no.  But I have yet to see
anything better.

> 
> - Given all you need is just to have a way to get the information about
> EAPI before you actually parse the ebuild since the eapi defines how you
> parse it, you can come up with various solutions, the simplest being
> first extract the eapi, being it in a fixed place, and then do the parse.
> 

Yes exactly,  you need to know the EAPI before you __parse__ the ebuild.
 At least we agree that nothing should have to read the contents of the
file to determine EAPI (doing so would be parsing now wouldn't it).  So
seeing that we agree with that,  where should we stick the EAPI.
mmmmm....

1)  How about in a flat txt file:   That would become a developers
nightmare.
2)  In an xml file.  Package managers would have to support xml.  Not
the best thing in the world. also could be a nightmare,  adding an entry
for every ebuild.
3)  As an xattr.  Well this idea is novel.  I bet it would make the tree
nice and stable too.  Lets not forget how annoying it will be for devs.
4) Parsing the ebuild.  But what are we parsing?,  lets not limit
ourselves to bash,  we might want to change languages completely.  If it
is bash,  what version, what if EAPI is set multiple times,  what if its
set in an eclass.
5)  Mmmmm...On the file name sounds like a good idea.  especially as an
extension.  provides information to a package manager, person,
script/program without them needing to open anything.  identifies the
contents just like .txt, .c, .o, .jpeg, etc

> - Extracting such information could have different costs depending on
> where to place it.

I believe it being on the filename would be the least costly,  in terms
of processor/io at least.

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Alistair Bush wrote:
> 
> Luca Barbato wrote:
>> Alistair Bush wrote:
>>> I just don't think those numbers tell us anything and that should be
>>> obvious from anyone who has read GLEP 55[1],  we ain't really attempting
>>> to solve a problem that exists within the tree currently (well the bash
>>> issue does, in a way ). We are trying to solve issues that ware stopping
>>> the "tree" moving forward.  Lets evaluate GLEP 55 in the problems it is
>>> attempting to solve.
>> I'm afraid you missed the whole point...
>>
>> - what is in the proposal is a solution looking for a problem: nobody
>> updated the glep with the required sections, nobody put up a list of
>> bugs/rfe from bugzilla it helps to solve. Vague "leading to the future
>> change" declaration aren't what I'd expect.
>>
> 
> So im mean't to start committing ebuilds into the tree that expect a
> certain unimplemented functionality, only to file bugs against portage
> for it not supporting them?

Apparently you missed rfe or that it does mean =\

> Plus we already know of at least one case where we will encounter a
> problem in the future,  why?  because we have already.
> Not sure if there is a open bug about it tho.

Do you know that "a problem" means nothing, bug #number means something?
Do you know that improvement and enhancement can be requested on 
bugzilla as well?

> This actually eats at me,  your basically saying GLEP's should only be
> reactive.  Why don't we all just roll over and die.

I'm afraid you are getting quite emotional for no reason.

> I have already considered the problems, and believe GLEP 55 is the
> **best** solution to them.  Is it perfect, no.  But I have yet to see
> anything better.

YOU, other did and consider what is proposed on that trash. Mediation is 
needed apparently. What is sure is that the glep proposal is lacking 
lots of details and apparently none of the proponents are even willing 
to try to cope with that.

> Yes exactly,  you need to know the EAPI before you __parse__ the ebuild.

You have to extract the eapi before doing the parsing the eapi defines, 
but you can parse the ebuild just to get the eapi and then do something 
or nothing depending on that value...

> 1)  How about in a flat txt file:   That would become a developers
> nightmare.
> 2)  In an xml file.  Package managers would have to support xml.  Not
> the best thing in the world. also could be a nightmare,  adding an entry
> for every ebuild.
> 3)  As an xattr.  Well this idea is novel.  I bet it would make the tree
> nice and stable too.  Lets not forget how annoying it will be for devs.
> 4) Parsing the ebuild.  But what are we parsing?,  lets not limit
> ourselves to bash,  we might want to change languages completely.  If it
> is bash,  what version, what if EAPI is set multiple times,  what if its
> set in an eclass.

"What if" is exactly something you cannot use, it's a slippery slope 
that leads to qbits frozen objects from the outher space or other insane 
stuff that may or may not happen.

> 5)  Mmmmm...On the file name sounds like a good idea.  especially as an
> extension.  provides information to a package manager, person,
> script/program without them needing to open anything.  identifies the
> contents just like .txt, .c, .o, .jpeg, etc

So for normal multimedia file I'd have to have "myfile.mov-aac-h264-ass" 
  as extension? strange, mplayer is perfectly fine even if it is called 
"myfile" and file(1) usually can tell me what's inside it in term of 
codec and sometimes even it's params.

>> - Extracting such information could have different costs depending on
>> where to place it.
> 
> I believe it being on the filename would be the least costly,  in terms
> of processor/io at least.

try yourself, I did and that's what I found regarding the case of cache 
regen (that, as I wrote earlier, is the interesting case) is in one of 
the previous emails as well...

btw it's also quite easy plant both proposals in portage and see what 
happen if you really like, but I preferred give something everybody can 
try in bash.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ferris McCormick]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

On Mon, 2009-02-23 at 22:19 +0000, Ciaran McCreesh wrote:
> On Mon, 23 Feb 2009 16:15:25 -0600
> Ryan Hill <dirtyepic@gentoo.org> wrote:
> > Can we ban eclasses from setting EAPI?  Is there any case where it
> > would be sane?
> 
> It's already banned from a QA perspective, but from a package manager
> perspective people have done it in the past and possibly still do do
> it, and the spec doesn't forbid it.
> 

For what it's worth, no eclass in the gentoo-x86/eclass tree sets EAPI.
I don't know about anyplace else.

Regards,
Ferris
-- 
Ferris McCormick (P44646, MI) <fmccor@gentoo.org>
Developer, Gentoo Linux (Sparc, Userrel, Trustees)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Serkan Kaba]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

2009/2/24 Ferris McCormick <fmccor@gentoo.org>

> On Mon, 2009-02-23 at 22:19 +0000, Ciaran McCreesh wrote:
> > On Mon, 23 Feb 2009 16:15:25 -0600
> > Ryan Hill <dirtyepic@gentoo.org> wrote:
> > > Can we ban eclasses from setting EAPI?  Is there any case where it
> > > would be sane?
> >
> > It's already banned from a QA perspective, but from a package manager
> > perspective people have done it in the past and possibly still do do
> > it, and the spec doesn't forbid it.
> >
>
> For what it's worth, no eclass in the gentoo-x86/eclass tree sets EAPI.
> I don't know about anyplace else.
>
> Regards,
> Ferris
> --
> Ferris McCormick (P44646, MI) <fmccor@gentoo.org>
> Developer, Gentoo Linux (Sparc, Userrel, Trustees)
>
lucene-contrib eclass in java-experimental [1] sets EAPI to 1 to use slot
deps. And I think that's a valid usage.

1:
http://overlays.gentoo.org/proj/java/browser/java-experimental/eclass/lucene-contrib.eclass

[-- Attachment #2: Type: text/html, Size: 1578 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

On Tue, 24 Feb 2009 08:08:23 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> Is there any technical merit in putting eapi in the file extension
> while we could restrict the format the same way in file and have
> about the same, negligible, performance hit? (I used warm cache since
> you need the file anyway so you don't spend time to look it up twice
> or put it in cache twice)

Uh, your benchmarks are nonsense. That is not how metadata checks work.
By parsing the ebuilds you're talking doubling the number of file reads
required to get the job done, and massively increasing the number of
seeks required.

But that isn't even the main issue. The main issue is that even if you
retroactively pretend that all ebuilds are in a format they're not, and
ignore the breakage, and then wait for a year for package managers to
try to parse your new format, you *still* can't change name or
versioning rules.

Again, these are all things that have been discussed at length
previously. Please either come up with a legitimate technical
objection, or admit that you've seen the light.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

On Mon, 23 Feb 2009 20:49:02 -0500
Richard Freeman <rich0@gentoo.org> wrote:
> > and it doesn't let you change name or version rules.
> 
> Neither does putting the EAPI in the filename as far as I can tell.

Name or versioning rules are things like allowing new suffixes or
altering the restrictions upon formatting.

Currently, package managers assume that they can handle anything named
pkg-$anything.ebuild, and $anything has to be a valid version spec by
the current rules.

Version rules have changed at least twice in the past, and it's been
messy every time. 55 allows version changes to be done safely (although
not carelessly...).

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Richard Freeman]

Alistair Bush wrote:
> 4) Parsing the ebuild.  But what are we parsing?,  lets not limit
> ourselves to bash,  we might want to change languages completely.  If it
> is bash,  what version, what if EAPI is set multiple times,  what if its
> set in an eclass.

How do you do this if you're getting EAPI from the filename?  How do you 
set it multiple times?  How do you set it in an eclass if you're getting 
it from the filename?

It seems like when we're talking about just putting the EAPI in a 
comment line on line x of the ebuild we're barraged with 47 ways that it 
will limit us, but when we're talking about EAPI in the filename 
suddenly we're not concerned with those limitations.  If it helps maybe 
we need to split EAPI into two records - one that deals with how to 
fundamentally parse the file and find out the EAPI, and the other that 
implements everything else the EAPI does.

I will certainly concede that putting it inside the ebuild potentially 
breaks compatibility with existing package managers.  That is certainly 
a downside to this approach.  However, none of the other objections that 
have been raised appear to hold water.  An EAPI in a filename is a blob 
of text that needs to be parsed out in one particular way with one set 
of system calls.  An EAPI embedded in the file is a blob of text that 
needs to be parsed out in one particular way with one set of system calls.

And if backwards compatibility were a serious issue you could define a 
new ".ebuild2" file spec that incorporates the EAPI inside the file and 
current package managers would ignore it.  Then you're not changing the 
file extension every time a new EAPI comes along, and the need to do so 
could be handled via future GLEPs.  Or we could just delay 
implementation and clean up existing package managers and tell users to 
migrate.

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

On Tue, 24 Feb 2009 10:46:30 -0500
Richard Freeman <rich0@gentoo.org> wrote:
> I will certainly concede that putting it inside the ebuild
> potentially breaks compatibility with existing package managers.
> That is certainly a downside to this approach.  However, none of the
> other objections that have been raised appear to hold water.

...and it means we can't change name or version rules.

...and it means over doubling the best possible time to work out a
dependency tree in the common case where the metadata cache is valid.

...and it means we can't make arbitrary format changes.

> And if backwards compatibility were a serious issue you could define
> a new ".ebuild2" file spec that incorporates the EAPI inside the file
> and current package managers would ignore it.  Then you're not
> changing the file extension every time a new EAPI comes along, and
> the need to do so could be handled via future GLEPs.

Developers already have to stop and think and consult the conveniently
available table of features for EAPIs. By splitting the EAPI concept in
two you're doubling the amount of data to be learnt.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Tue, 24 Feb 2009 08:08:23 +0100
> Uh, your benchmarks are nonsense.

Provide your nonsensical ones.

> That is not how metadata checks work.

Explain how they work, regen works that way...

> By parsing the ebuilds you're talking doubling the number of file reads
> required to get the job done, and massively increasing the number of
> seeks required.

Apparently it doesn't impact anything.

> But that isn't even the main issue. The main issue is that even if you
> retroactively pretend that all ebuilds are in a format they're not, and
> ignore the breakage, and then wait for a year for package managers to
> try to parse your new format, you *still* can't change name or
> versioning rules.

why? when portage would breanch if I put an ebuild with a wacky version 
AND there is a valid cache for that telling its eapi 99 ?

> Again, these are all things that have been discussed at length
> previously. Please either come up with a legitimate technical
> objection, or admit that you've seen the light.

the glep doesn't show any of those nor reference to it, as I said 
before, do your homework and probably more people will be happier with 
your proposals.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2398 bytes --]

On Tue, 24 Feb 2009 17:04:28 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > On Tue, 24 Feb 2009 08:08:23 +0100
> > Uh, your benchmarks are nonsense.
> 
> Provide your nonsensical ones.

You're doubling the number of files that have to be read for an
operation that's almost purely i/o bound. On top of that, you're
introducing a whole bunch of disk seeks in what's otherwise a nice
linear operation.

> > That is not how metadata checks work.
> 
> Explain how they work, regen works that way...

If metadata is valid, ebuilds aren't opened at all. An optimal
implementation can slurp up the entire directory in one go and then
start pulling out cache entries as it needs them, not having to go back
to the ebuild directory or read its contents at all. Then it can open
and read cache files in a carefully selected order to avoid having to
do any more opens than necessary.

> > By parsing the ebuilds you're talking doubling the number of file
> > reads required to get the job done, and massively increasing the
> > number of seeks required.
> 
> Apparently it doesn't impact anything.

Please show the patch you created (for Paludis, since Portage doesn't
yet do a lot of the optimisations it could here) that demonstrates this.

> > But that isn't even the main issue. The main issue is that even if
> > you retroactively pretend that all ebuilds are in a format they're
> > not, and ignore the breakage, and then wait for a year for package
> > managers to try to parse your new format, you *still* can't change
> > name or versioning rules.
> 
> why? when portage would breanch if I put an ebuild with a wacky
> version AND there is a valid cache for that telling its eapi 99 ?

Because it has to parse that version. Also, the package manager can't
tell whether or not a cache entry is valid if it doesn't recognise the
EAPI in the cache entry.

> > Again, these are all things that have been discussed at length
> > previously. Please either come up with a legitimate technical
> > objection, or admit that you've seen the light.
> 
> the glep doesn't show any of those nor reference to it, as I said 
> before, do your homework and probably more people will be happier
> with your proposals.

Why should it? The C++ standard doesn't explain why you should use it
instead of Java...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Joe Peterson]

Richard Freeman wrote:
> I still don't see why we need to be encoding metadata in filenames.

Correct.  GLEP 55 tries to solve a technical implementation issue by
exposing meta data in the filename.  Extremely bad form/design, IMHO.

> PERL doesn't care what a file extension is, python doesn't care, bzip2 
> doesn't care, tar doesn't care, gzip doesn't care, and even ld-linux.so 
> doesn't care.  I'm sure that in at least some of these cases they end up 
> parsing parts of the file twice - once to figure out what it is, and the 
> second time to actually handle it.  I'm actually hard pressed to think 
> of any unix-based software that uses the filename to store a mandatory 
> file format versioning specifier of some kind.

All good points.  I cannot believe there exists no other way to solve
this technical issue other than resorting to such a non-Unix-like idea.
Obviously all of the software packages cited above endeavor to avoid
such nastiness.

I do not understand why anyone is willing to accept putting version info
in the filename/extension.  It is inelegant and, frankly, very ugly.  I
have written more in the past on why I think it is a terrible idea, so I
won't repeat it here.

Suffice to say, if something like GLEP 55 is implemented, I will lose a
lot of faith in Gentoo's design, so much so that I will likely join the
ranks of those who abandon it, not only as a dev, but also as a user.

> This seems to me to be a solved problem.  You put a header in a file 
> that tells you how to read the file.  Headers are split into fields and 
> if a program doesn't know what to do with a field it ignores it (or if 
> the header so instructs it doesn't even try to parse the file).  This 
> should be easy to do and keep the file bash-compatible.  Just stick a 
> comment line close to the top of the file and put whatever you want on 
> it.  You could also stick something in metadata.xml (although this makes 
> working with ebuilds outside of a repository more difficult).  You run 
> the file through an algorithm to find out what the EAPI is, and then 
> source it if appropriate.
> 
> Sure, if you make some major change analogous to switching from the .rpm 
> to the .deb package format then maybe an extension change would make 
> sense.  But, why expose the inner workings of the package file format to 
> the filesystem?

+100

						-Joe

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alec Warner]

Somewhat ironically, had everyone been less stubborn last year when
discussing this topic we could have embedded the EAPI in line X of the
ebuild in 2008 and be using it now; instead of still discussing it.

I don't expect new novel ideas out of this thread.  I expect the
council to defer it again because the arguments are the same as last
time and last time they were not convincing enough.  I would prefer if
the council went one way or the other so that when we are arguing
about this in 2010 we can at least say "hey we have support in
$PACKAGE_MANAGERs for EAPI on line X since May 2009 so in 3 months we
can just switch.  We don't have to make the switch; I'm just saying we
should add support to hedge our bets.

Thoughts?

-A

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1478 bytes --]

On Tue, 24 Feb 2009 09:15:59 -0700
Joe Peterson <lavajoe@gentoo.org> wrote:
> Richard Freeman wrote:
> > I still don't see why we need to be encoding metadata in filenames.
> 
> Correct.  GLEP 55 tries to solve a technical implementation issue by
> exposing meta data in the filename.  Extremely bad form/design, IMHO.

We already expose metadata in the filename. The version's there and the
name's there.

> All good points.  I cannot believe there exists no other way to solve
> this technical issue other than resorting to such a non-Unix-like
> idea. Obviously all of the software packages cited above endeavor to
> avoid such nastiness.

Then why don't you come up with a viable solution?

> I do not understand why anyone is willing to accept putting version
> info in the filename/extension.  It is inelegant and, frankly, very
> ugly.  I have written more in the past on why I think it is a
> terrible idea, so I won't repeat it here.

For the same reason they're willing to accept the package name and
version in the filename.

> Suffice to say, if something like GLEP 55 is implemented, I will lose
> a lot of faith in Gentoo's design, so much so that I will likely join
> the ranks of those who abandon it, not only as a dev, but also as a
> user.

"If you paint the bikeshed, I shall throw my toys out of the pram and
run off crying.".

Why don't you propose a viable alternative instead of making threats?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 434 bytes --]

On Tue, 24 Feb 2009 08:21:25 -0800
Alec Warner <antarus@gentoo.org> wrote:
> Somewhat ironically, had everyone been less stubborn last year when
> discussing this topic we could have embedded the EAPI in line X of the
> ebuild in 2008 and be using it now; instead of still discussing it.

...and we wouldn't be able to change the version rules, and we would be
suffering a substantial performance hit.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Joe Peterson]

Richard Freeman wrote:
> The dynamic linker doesn't need to consult the filename to figure out 
> how to parse a shared object.  It only consults the filename to figure 
> out which shared object is needed.  That is actually analogous to 
> putting the package name/version in the ebuild filename.

Right.  Plus, if the linker *did* consult the filename, imagine what
would happen if someone renamed the file (even by accident) and changed
the version?  The parser should not be able to be so easily fooled -
could cause great confusion and or nasty and weird bugs - seems very
fragile to me.  Having the version *in* the file is much safer, since
monkeying with that would require editing it the file, rather than
renaming it.

						-Joe

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Nirbheek Chauhan]

On Tue, Feb 24, 2009 at 9:26 PM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> ...and it means we can't change name or version rules.
>

And has such a case come to light recently where it was *essential* to
do so? Why solve problems that don't exist?

> ...and it means over doubling the best possible time to work out a
> dependency tree in the common case where the metadata cache is valid.
>

This is a valid cause. Perhaps the only valid cause.

> ...and it means we can't make arbitrary format changes.
>

What? Why are we over-engineering this? Does anyone seriously want to
convert ebuilds to XML? I honestly think anything beyond incremental
changes is not relevant for Gentoo

> Developers already have to stop and think and consult the conveniently
> available table of features for EAPIs. By splitting the EAPI concept in
> two you're doubling the amount of data to be learnt.
>

That's a documentation problem.


-- 
~Nirbheek Chauhan

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

On Tue, 24 Feb 2009 09:24:48 -0700
Joe Peterson <lavajoe@gentoo.org> wrote:
> Right.  Plus, if the linker *did* consult the filename, imagine what
> would happen if someone renamed the file (even by accident) and
> changed the version?  The parser should not be able to be so easily
> fooled - could cause great confusion and or nasty and weird bugs -
> seems very fragile to me.  Having the version *in* the file is much
> safer, since monkeying with that would require editing it the file,
> rather than renaming it.

You could use the same absurd argument to say that PN and PV shouldn't
be in the filename...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alec Warner]

On Tue, Feb 24, 2009 at 8:23 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> On Tue, 24 Feb 2009 08:21:25 -0800
> Alec Warner <antarus@gentoo.org> wrote:
>> Somewhat ironically, had everyone been less stubborn last year when
>> discussing this topic we could have embedded the EAPI in line X of the
>> ebuild in 2008 and be using it now; instead of still discussing it.
>
> ...and we wouldn't be able to change the version rules, and we would be
> suffering a substantial performance hit.

Hey I never said its a perfect solution; but I'm a fan of the 'it
covers 80%'.   Sometimes you can't have your cake and eat it too;
sometimes requirements get dropped.

-A

>
> --
> Ciaran McCreesh
>

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Joe Peterson]

Ciaran McCreesh wrote:
> On Tue, 24 Feb 2009 09:24:48 -0700
> Joe Peterson <lavajoe@gentoo.org> wrote:
>> Right.  Plus, if the linker *did* consult the filename, imagine what
>> would happen if someone renamed the file (even by accident) and
>> changed the version?  The parser should not be able to be so easily
>> fooled - could cause great confusion and or nasty and weird bugs -
>> seems very fragile to me.  Having the version *in* the file is much
>> safer, since monkeying with that would require editing it the file,
>> rather than renaming it.
> 
> You could use the same absurd argument to say that PN and PV shouldn't
> be in the filename...

No...!

They are needed because:

1) versions of the *content*, not the *format* are needed for uniqueness
2) it makes sense to have these in the filename, but not internal meta-data

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 376 bytes --]

On Tue, 24 Feb 2009 08:33:19 -0800
Alec Warner <antarus@gentoo.org> wrote:
> Hey I never said its a perfect solution; but I'm a fan of the 'it
> covers 80%'.   Sometimes you can't have your cake and eat it too;
> sometimes requirements get dropped.

We can cover 100% with a solution with less of an impact. There's no
need to compromise here.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alec Warner]

On Tue, Feb 24, 2009 at 8:37 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> On Tue, 24 Feb 2009 08:33:19 -0800
> Alec Warner <antarus@gentoo.org> wrote:
>> Hey I never said its a perfect solution; but I'm a fan of the 'it
>> covers 80%'.   Sometimes you can't have your cake and eat it too;
>> sometimes requirements get dropped.
>
> We can cover 100% with a solution with less of an impact. There's no
> need to compromise here.

Two years to argue and implement a solution certainly shouts "compromise" to me.

-A

>
> --
> Ciaran McCreesh
>

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

On Tue, 24 Feb 2009 09:36:29 -0700
Joe Peterson <lavajoe@gentoo.org> wrote:
> > You could use the same absurd argument to say that PN and PV
> > shouldn't be in the filename...
> 
> No...!
> 
> They are needed because:
> 
> 1) versions of the *content*, not the *format* are needed for
> uniqueness

So why's PN in there then?

> 2) it makes sense to have these in the filename, but not
> internal meta-data

For those of us who understand the process, it makes sense to have EAPI
in the filename too.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Joe Peterson]

Ciaran McCreesh wrote:
>> All good points.  I cannot believe there exists no other way to solve
>> this technical issue other than resorting to such a non-Unix-like
>> idea. Obviously all of the software packages cited above endeavor to
>> avoid such nastiness.
> 
> Then why don't you come up with a viable solution?

I already have - look back at my posts; very similar to Rich0's idea.
And I tire of the argument that if one doesn't have a perfect solution
now, we should adopt a half-brained one.  The point of this is to spur
discussion to come up with a better solution.

> For the same reason they're willing to accept the package name and
> version in the filename.

The fact that you think this is the same thing as having the EAPI in the
filename is odd.

> "If you paint the bikeshed, I shall throw my toys out of the pram and
> run off crying.".
> 
> Why don't you propose a viable alternative instead of making threats?

Not a threat.  And this will be my last post on the topic.  I will not
take your bate and continue to argue, creating more noise on this list -
I've expressed how I feel.

							-Joe

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1417 bytes --]

On Tue, 24 Feb 2009 21:59:39 +0530
Nirbheek Chauhan <nirbheek@gentoo.org> wrote:
> On Tue, Feb 24, 2009 at 9:26 PM, Ciaran McCreesh
> <ciaran.mccreesh@googlemail.com> wrote:
> > ...and it means we can't change name or version rules.
> 
> And has such a case come to light recently where it was *essential* to
> do so? Why solve problems that don't exist?

Because they do exist, which is why name and version rules have been
changed the hard way at least twice previously. The version format is
still considerably less flexible than what upstreams use, and a lot of
the current limitations on its format are purely historical.

> > ...and it means we can't make arbitrary format changes.
> 
> What? Why are we over-engineering this? Does anyone seriously want to
> convert ebuilds to XML? I honestly think anything beyond incremental
> changes is not relevant for Gentoo

You appear to be confusing arbitrary format changes with doing a Zynot.
The two are not the same.

> > Developers already have to stop and think and consult the
> > conveniently available table of features for EAPIs. By splitting
> > the EAPI concept in two you're doubling the amount of data to be
> > learnt.
> 
> That's a documentation problem.

No, it's a design problem. Good design looks for ways to minimise the
amount of unnecessary arbitrary information the user has to remember.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1230 bytes --]

On Tue, 24 Feb 2009 09:45:44 -0700
Joe Peterson <lavajoe@gentoo.org> wrote:
> > Then why don't you come up with a viable solution?
> 
> I already have - look back at my posts; very similar to Rich0's idea.

No, I said viable.

> And I tire of the argument that if one doesn't have a perfect solution
> now, we should adopt a half-brained one.  The point of this is to spur
> discussion to come up with a better solution.

We have a perfect solution.

> > For the same reason they're willing to accept the package name and
> > version in the filename.
> 
> The fact that you think this is the same thing as having the EAPI in
> the filename is odd.

PN and PV are metadata, same as EAPI.

> > "If you paint the bikeshed, I shall throw my toys out of the pram
> > and run off crying.".
> > 
> > Why don't you propose a viable alternative instead of making
> > threats?
> 
> Not a threat.  And this will be my last post on the topic.  I will not
> take your bate and continue to argue, creating more noise on this
> list - I've expressed how I feel.

This isn't about how you feel. It's about what you rationally think,
based upon a full understanding of the issues at hand.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

On Tue, 24 Feb 2009 08:42:44 -0800
Alec Warner <antarus@gentoo.org> wrote:
> On Tue, Feb 24, 2009 at 8:37 AM, Ciaran McCreesh
> <ciaran.mccreesh@googlemail.com> wrote:
> > On Tue, 24 Feb 2009 08:33:19 -0800
> > Alec Warner <antarus@gentoo.org> wrote:
> >> Hey I never said its a perfect solution; but I'm a fan of the 'it
> >> covers 80%'.   Sometimes you can't have your cake and eat it too;
> >> sometimes requirements get dropped.
> >
> > We can cover 100% with a solution with less of an impact. There's no
> > need to compromise here.
> 
> Two years to argue and implement a solution certainly shouts
> "compromise" to me.

Strange. To me it shouts "leadership problem".

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Tue, 24 Feb 2009 17:04:28 +0100
> Luca Barbato <lu_zero@gentoo.org> wrote:
>> Ciaran McCreesh wrote:
>>> On Tue, 24 Feb 2009 08:08:23 +0100
>>> Uh, your benchmarks are nonsense.
>> Provide your nonsensical ones.
> 
> You're doubling the number of files that have to be read for an
> operation that's almost purely i/o bound. On top of that, you're
> introducing a whole bunch of disk seeks in what's otherwise a nice
> linear operation.

I see words, not numbers.

>>> That is not how metadata checks work.
>> Explain how they work, regen works that way...
> 
> If metadata is valid, ebuilds aren't opened at all. An optimal
> implementation can slurp up the entire directory in one go and then
> start pulling out cache entries as it needs them, not having to go back
> to the ebuild directory or read its contents at all. Then it can open
> and read cache files in a carefully selected order to avoid having to
> do any more opens than necessary.

So? if the cache is valid then you don't have to source them at all. If 
you have to regen it, well you have to read everything.

>>> By parsing the ebuilds you're talking doubling the number of file
>>> reads required to get the job done, and massively increasing the
>>> number of seeks required.
>> Apparently it doesn't impact anything.
> 
> Please show the patch you created (for Paludis, since Portage doesn't
> yet do a lot of the optimisations it could here) that demonstrates this.

Paludis isn't portage.

>>> But that isn't even the main issue. The main issue is that even if
>>> you retroactively pretend that all ebuilds are in a format they're
>>> not, and ignore the breakage, and then wait for a year for package
>>> managers to try to parse your new format, you *still* can't change
>>> name or versioning rules.
>> why? when portage would breanch if I put an ebuild with a wacky
>> version AND there is a valid cache for that telling its eapi 99 ?
>
> Because it has to parse that version. Also, the package manager can't
> tell whether or not a cache entry is valid if it doesn't recognise the
> EAPI in the cache entry.

unknown isn't unsupported?

>>> Again, these are all things that have been discussed at length
>>> previously. Please either come up with a legitimate technical
>>> objection, or admit that you've seen the light.
>> the glep doesn't show any of those nor reference to it, as I said 
>> before, do your homework and probably more people will be happier
>> with your proposals.
> 
> Why should it? The C++ standard doesn't explain why you should use it
> instead of Java...

In fact many people do wonderful things with java and many more just do
over engineered mess with C++?

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Jim Ramsay]

[-- Attachment #1: Type: text/plain, Size: 1636 bytes --]

Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> On Tue, 24 Feb 2009 10:46:30 -0500
> Richard Freeman <rich0@gentoo.org> wrote:
> > I will certainly concede that putting it inside the ebuild
> > potentially breaks compatibility with existing package managers.
> > That is certainly a downside to this approach.  However, none of the
> > other objections that have been raised appear to hold water.
> 
> ...and it means we can't change name or version rules.
> 
> ...and it means over doubling the best possible time to work out a
> dependency tree in the common case where the metadata cache is valid.
> 
> ...and it means we can't make arbitrary format changes.

Those would all land in the category of "backwards compatibility"
mentioned below, as they would all break current sourcing rules.

> > And if backwards compatibility were a serious issue you could define
> > a new ".ebuild2" file spec that incorporates the EAPI inside the
> > file and current package managers would ignore it.  Then you're not
> > changing the file extension every time a new EAPI comes along, and
> > the need to do so could be handled via future GLEPs.
> 
> Developers already have to stop and think and consult the conveniently
> available table of features for EAPIs. By splitting the EAPI concept
> in two you're doubling the amount of data to be learnt.
 
I would think that this is a very small cost, and the benefit would be
(I hope) that more people would agree on the solution and then we can go
forward. Is that not a valid consideration?

-- 
Jim Ramsay
Gentoo Developer (rox/fluxbox/gkrellm/vim)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1201 bytes --]

On Tue, 24 Feb 2009 12:25:27 -0500
Jim Ramsay <lack@gentoo.org> wrote:
> > ...and it means we can't change name or version rules.
> > 
> > ...and it means over doubling the best possible time to work out a
> > dependency tree in the common case where the metadata cache is
> > valid.
> > 
> > ...and it means we can't make arbitrary format changes.
> 
> Those would all land in the category of "backwards compatibility"
> mentioned below, as they would all break current sourcing rules.

No, they're also future issues. Without glep 55, every time they come
up we have to go through the whole mess again.

> > Developers already have to stop and think and consult the
> > conveniently available table of features for EAPIs. By splitting
> > the EAPI concept in two you're doubling the amount of data to be
> > learnt.
>  
> I would think that this is a very small cost, and the benefit would be
> (I hope) that more people would agree on the solution and then we can
> go forward. Is that not a valid consideration?

I'd expect to see changes that would warrant a major bump in every
other EAPI or so anyway, so it's not really worth the complexity.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3034 bytes --]

On Tue, 24 Feb 2009 18:16:54 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> > You're doubling the number of files that have to be read for an
> > operation that's almost purely i/o bound. On top of that, you're
> > introducing a whole bunch of disk seeks in what's otherwise a nice
> > linear operation.
> 
> I see words, not numbers.

Number: double. That's a '2 times'.

> >>> That is not how metadata checks work.
> >> Explain how they work, regen works that way...
> > 
> > If metadata is valid, ebuilds aren't opened at all. An optimal
> > implementation can slurp up the entire directory in one go and then
> > start pulling out cache entries as it needs them, not having to go
> > back to the ebuild directory or read its contents at all. Then it
> > can open and read cache files in a carefully selected order to
> > avoid having to do any more opens than necessary.
> 
> So? if the cache is valid then you don't have to source them at all.
> If you have to regen it, well you have to read everything.

You don't know whether the cache is valid until you know the EAPI. It
only works currently because EAPIs don't change inherit behaviour.

> >>> By parsing the ebuilds you're talking doubling the number of file
> >>> reads required to get the job done, and massively increasing the
> >>> number of seeks required.
> >> Apparently it doesn't impact anything.
> > 
> > Please show the patch you created (for Paludis, since Portage
> > doesn't yet do a lot of the optimisations it could here) that
> > demonstrates this.
> 
> Paludis isn't portage.

So you have patches for Portage? Please show them.

> >>> But that isn't even the main issue. The main issue is that even if
> >>> you retroactively pretend that all ebuilds are in a format they're
> >>> not, and ignore the breakage, and then wait for a year for package
> >>> managers to try to parse your new format, you *still* can't change
> >>> name or versioning rules.
> >> why? when portage would breanch if I put an ebuild with a wacky
> >> version AND there is a valid cache for that telling its eapi 99 ?
> >
> > Because it has to parse that version. Also, the package manager
> > can't tell whether or not a cache entry is valid if it doesn't
> > recognise the EAPI in the cache entry.
> 
> unknown isn't unsupported?

Huh? Please explain what you mean.

> >>> Again, these are all things that have been discussed at length
> >>> previously. Please either come up with a legitimate technical
> >>> objection, or admit that you've seen the light.
> >> the glep doesn't show any of those nor reference to it, as I said 
> >> before, do your homework and probably more people will be happier
> >> with your proposals.
> > 
> > Why should it? The C++ standard doesn't explain why you should use
> > it instead of Java...
> 
> In fact many people do wonderful things with java and many more just
> do over engineered mess with C++?

Your trolling is going rapidly downhill.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ryan Hill]

Alec Warner <antarus <at> gentoo.org> writes:

> Somewhat ironically, had everyone been less stubborn last year when
> discussing this topic we could have embedded the EAPI in line X of the
> ebuild in 2008 and be using it now; instead of still discussing it.
> 
> I don't expect new novel ideas out of this thread.  I expect the
> council to defer it again because the arguments are the same as last
> time and last time they were not convincing enough.  I would prefer if
> the council went one way or the other so that when we are arguing
> about this in 2010 we can at least say "hey we have support in
> $PACKAGE_MANAGERs for EAPI on line X since May 2009 so in 3 months we
> can just switch.  We don't have to make the switch; I'm just saying we
> should add support to hedge our bets.
> 
> Thoughts?

Well I give up.  There have been exactly zero technical arguments against GLEP 
55 and plenty of rhetoric, but if that's enough to sway people then so be it.  
If we take EAPI extensions off the table, which of these would work the best 
(aka. gives people the warm fuzzies).

- eapi in the file name, still ends in .ebuild
-- no parsing needed
-- doesn't allow version suffix additions/changes
-- requires an initial wait period for PM's to implement support and be 
stabilized
-- still makes some people grumpy/threaten to leave

- eapi in the ebuild, on a predetermined line number
-- error prone, restrictive
-- doesn't require parsing
-- doesn't allow version suffix additions/changes

- eapi in the ebuild, anywhere
-- what we have now
-- currently requires sourcing the ebuild
-- sourcing the ebuild requires knowing the EAPI
-- doesn't allow version suffix additions/changes (actually, none of these do, 
so i'll stop repeating it)

- eapi in the ebuild, before any other assignments/commands
-- ie. if we hit inherit and no EAPI is defined, EAPI=0
-- restrictive (eapi must be a direct assignment, no conditionals, etc)
-- no per category/PN eapi's or eapi's set in eclasses
-- probably the next best solution (IMUO)

- eapi in metadata somewhere else
-- meh, i'll let the proponents of this argue for it

please fill your arguments for or against, or fix whatever i have wrong.

some other random ideas i've seen tossed around:
- #!/bin/env eapi-parser
- split EAPI into EAPI and some separate counter which would only be 
incremented on uncompatible changes to the ebuild format
- change .ebuild to .eb
- waffles (sorry, lunchtime)

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Jim Ramsay]

[-- Attachment #1: Type: text/plain, Size: 3622 bytes --]

Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> On Tue, 24 Feb 2009 12:25:27 -0500
> Jim Ramsay <lack@gentoo.org> wrote:
> > > ...and it means we can't change name or version rules.
> > > 
> > > ...and it means over doubling the best possible time to work out a
> > > dependency tree in the common case where the metadata cache is
> > > valid.
> > > 
> > > ...and it means we can't make arbitrary format changes.
> > 
> > Those would all land in the category of "backwards compatibility"
> > mentioned below, as they would all break current sourcing rules.
> 
> No, they're also future issues. Without glep 55, every time they come
> up we have to go through the whole mess again.

This minor/major EAPI scheme is exactly equivalent to glep 55 in
the ways that it addresses the non-implementation-specific
details - It could even be added as a caveat to glep-55 that says
something like:

"You should not change filename extension (ie, major-EAPI version, or
EPARSE version, or whatever we want to call it...) except when you *have
to* because of changes such as name or version rules, arbitrary format
changes, or anything else that breaks the sourcing rules of the
existing filename extensions. Simpler feature improvements can be done
using whatever internal minor-EAPI version is defined by the major-EAPI
version."

This doesn't prevent you from changing the filename extension when you
have to do so, it just suggests that maybe you don't have to do it for
every single feature-set you may want to implement.

> > > Developers already have to stop and think and consult the
> > > conveniently available table of features for EAPIs. By splitting
> > > the EAPI concept in two you're doubling the amount of data to be
> > > learnt.
> >  
> > I would think that this is a very small cost, and the benefit would
> > be (I hope) that more people would agree on the solution and then
> > we can go forward. Is that not a valid consideration?
> 
> I'd expect to see changes that would warrant a major bump in every
> other EAPI or so anyway, so it's not really worth the complexity.

If that is indeed the case, then adding this caveat to glep-55 is
basically a nop.  If every EAPI includes a non-backwards-compatible
change that breaks existing PMs, the filename extension will be changed
every time.

But when you say "worth the complexity", I'm not exactly sure what
your standards of "worthiness" are.

I don't think the human cognition of a 2-level versioning scheme is
that tricky, so I assume you must mean complexity in the internals of
package managers - but this should just be a Simple Matter Of
Programming.

I'll further qualify this response by mentioning that I am not a package
manager maintainer.  I don't know beans about metadata and cacheing and
what the tradeoffs may be between a two-level EAPI and a single-level
EAPI stored in the filename.  I understand that parsing two-level EAPI
is "more expensive" than a single-level stored in the filename.  I don't
however know how to figure out if it is "too expensive", or whose
subjective scale we should use to measure this.

I personally feel the "complexity" that you say is too costly is a fair
tradeoff for a proposal that people will accept.

(Of course I have no idea if people actually would accept a two-level
EAPI any more than glep-55 as-is... I just think it addresses the
concerns I've heard in this thread in a way that does not break
the valid solutions to real problems presented in glep-55)

-- 
Jim Ramsay
Gentoo Developer (rox/fluxbox/gkrellm/vim)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1350 bytes --]

On Tue, 24 Feb 2009 14:08:45 -0500
Jim Ramsay <lack@gentoo.org> wrote:
> But when you say "worth the complexity", I'm not exactly sure what
> your standards of "worthiness" are.
> 
> I don't think the human cognition of a 2-level versioning scheme is
> that tricky, so I assume you must mean complexity in the internals of
> package managers - but this should just be a Simple Matter Of
> Programming.

People are struggling with the one level scheme we have now. We're
already having to produce fancy tables and summaries for new EAPIs
because people can't keep track of when features came along. Two levels
just means no-one will remember any of it.

For the package manager, it's just a bit of added mess, not any major
difficulty.

> (Of course I have no idea if people actually would accept a two-level
> EAPI any more than glep-55 as-is... I just think it addresses the
> concerns I've heard in this thread in a way that does not break
> the valid solutions to real problems presented in glep-55)

People are opposed to 55 because of a knee-jerk reaction against
changing file extensions and against doing anything that comes from
the great Satan and all his little minions... If you're going to throw
an equivalent but supposedly compromising solution at people, go for
'.eapi3.eb' instead.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 925 bytes --]

On Tue, 24 Feb 2009 18:24:16 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Tue, 24 Feb 2009 18:16:54 +0100
> Luca Barbato <lu_zero@gentoo.org> wrote:
> > > You're doubling the number of files that have to be read for an
> > > operation that's almost purely i/o bound. On top of that, you're
> > > introducing a whole bunch of disk seeks in what's otherwise a nice
> > > linear operation.
> > 
> > I see words, not numbers.
> 
> Number: double. That's a '2 times'.

That only means you're increasing the constant factor in the
complexity of the thing... which may very well be completely negligible
unless someone provides real benchmarks. I would be very surprised if
that "2 times" factor happens to be true, because finding a string in a
file is an order of magnitude simpler than sourcing said file with
bash. Moreover this doesn't take into account disk and os cache.

Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1336 bytes --]

On Tue, 24 Feb 2009 20:28:43 +0100
Alexis Ballier <aballier@gentoo.org> wrote:
> On Tue, 24 Feb 2009 18:24:16 +0000
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > On Tue, 24 Feb 2009 18:16:54 +0100
> > Luca Barbato <lu_zero@gentoo.org> wrote:
> > > > You're doubling the number of files that have to be read for an
> > > > operation that's almost purely i/o bound. On top of that, you're
> > > > introducing a whole bunch of disk seeks in what's otherwise a
> > > > nice linear operation.
> > > 
> > > I see words, not numbers.
> > 
> > Number: double. That's a '2 times'.
> 
> That only means you're increasing the constant factor in the
> complexity of the thing... which may very well be completely
> negligible unless someone provides real benchmarks.

In the most common case where metadata is valid, around half of the
time it takes Paludis to work out a resolution set is spent grabbing
metadata for ebuilds.

> I would be very surprised if that "2 times" factor happens to be true,
> because finding a string in a file is an order of magnitude simpler
> than sourcing said file with bash. Moreover this doesn't take into
> account disk and os cache.

No no no. *Opening* the file is the slow part, not searching. The file
wouldn't otherwise be opened at all.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Jim Ramsay]

[-- Attachment #1: Type: text/plain, Size: 2552 bytes --]

Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> People are struggling with the one level scheme we have now. We're
> already having to produce fancy tables and summaries for new EAPIs
> because people can't keep track of when features came along. Two
> levels just means no-one will remember any of it.

I disagree with your assertion that people are struggling - I think
things are very nicely documented in PMS and the devmanual, which are
where all EAPI changes should be documented in the future, regardless
if you specify the EAPI in the file, the extension, or both.

Two levels really just means that any fancy tables will have to have
one extra row (or perhaps a series of fancy tables) and any summaries
will have to have an extra section added whenever a new filename
extension becomes necessary.

> For the package manager, it's just a bit of added mess, not any major
> difficulty.

This is good to know, thank you for the clarification.

> People are opposed to 55 because of a knee-jerk reaction against
> changing file extensions and against doing anything that comes from
> the great Satan and all his little minions... If you're going to throw
> an equivalent but supposedly compromising solution at people, go for
> '.eapi3.eb' instead.

I can't speak to anyone's motivations or religious beliefs other than my
own here, but the opposition I have heard most often in this thread
is something like: "I don't like it when the file extension changes so
often". Some people site historical president or the way other software
does things, or whatever -> doesn't really matter.

What does matter is that some people don't like it when the file
extension changes very often. I think my solution is a valid compromise
because it balances, in my opinion, the two camps, whose arguments I
summarize as:

glep-55'ers: "I don't care if the file extension changes all the time, I
just want a solution that works and is reasonably future-proof"

Anti-55'ers: "I don't want the file extension to change ever, but I
would agree that for major-enough changes it may be required
sometimes"

If I understand the '.eapi3.eb' to which you make passing reference,
this is just a fancy hand-wavy way to say "Look, the true .eb
extension won't ever change, just the .eapi3 part which isn't
technically the extension..." which isn't a compromise at all - It's an
attempt to (cleverly?) obfuscate where in the filename the EAPI is
stored.

-- 
Jim Ramsay
Gentoo Developer (rox/fluxbox/gkrellm)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

On Tue, 24 Feb 2009 15:07:29 -0500
Jim Ramsay <lack@gentoo.org> wrote:
> Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > People are struggling with the one level scheme we have now. We're
> > already having to produce fancy tables and summaries for new EAPIs
> > because people can't keep track of when features came along. Two
> > levels just means no-one will remember any of it.
> 
> I disagree with your assertion that people are struggling - I think
> things are very nicely documented in PMS and the devmanual, which are
> where all EAPI changes should be documented in the future, regardless
> if you specify the EAPI in the file, the extension, or both.

They only ended up nicely documented after people moaned a lot that
they were having a hard time keeping track of EAPIs...

> Two levels really just means that any fancy tables will have to have
> one extra row (or perhaps a series of fancy tables) and any summaries
> will have to have an extra section added whenever a new filename
> extension becomes necessary.

It'll mean people will carry on having to use the tables, and won't
start remembering things as time goes on.

> If I understand the '.eapi3.eb' to which you make passing reference,
> this is just a fancy hand-wavy way to say "Look, the true .eb
> extension won't ever change, just the .eapi3 part which isn't
> technically the extension..." which isn't a compromise at all - It's
> an attempt to (cleverly?) obfuscate where in the filename the EAPI is
> stored.

Yup. And yet there're people who are perfectly happy with .eapi3.eb who
hate GLEP 55. That should tell you all you need to know about what's
going on here...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Richard Freeman]

Ciaran McCreesh wrote:
> 
> ..and it means we can't change name or version rules.
> 

Why?  Just parse the EAPI out of the file before you interpret the 
version and name from the filename.  Indeed - you could have a future 
EAPI remove the name and version from the filename entirely.  If a 
package manager doesn't understand the EAPI in a file it shouldn't do 
anything at all with it.

> ..and it means over doubling the best possible time to work out a
> dependency tree in the common case where the metadata cache is valid.
> 

I can see why it takes an extra pass - but does that mean a doubling of 
time?  Couldn't the EAPI be cached as well to reduce disk access?

> ..and it means we can't make arbitrary format changes.

Well, you would need to preserve the EAPI in the header, but other than 
that you could actually turn an ebuild into an otherwise completely 
binary file, or whatever.  Just how much more flexibility than that is 
needed?

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1717 bytes --]

On Tue, 24 Feb 2009 15:17:01 -0500
Richard Freeman <rich0@gentoo.org> wrote:
> Why?  Just parse the EAPI out of the file before you interpret the 
> version and name from the filename.  Indeed - you could have a future 
> EAPI remove the name and version from the filename entirely.  If a 
> package manager doesn't understand the EAPI in a file it shouldn't do 
> anything at all with it.

Then you get into the mess of deciding what is or is not an ebuild...
Currently it's well defined; if you start making the package manager
look inside files things get very confusing...

> > ..and it means over doubling the best possible time to work out a
> > dependency tree in the common case where the metadata cache is
> > valid.
> 
> I can see why it takes an extra pass - but does that mean a doubling
> of time?  Couldn't the EAPI be cached as well to reduce disk access?

It means opening a file that would otherwise not be opened at all. And
if the EAPI is in the file, you have to fish inside that file to pull
it out before you can work out whether the cache entry that might
contain the EAPI already is valid.

(We don't have to do this currently because inherit hasn't changed
behaviour at all.)

> > ..and it means we can't make arbitrary format changes.
> 
> Well, you would need to preserve the EAPI in the header, but other
> than that you could actually turn an ebuild into an otherwise
> completely binary file, or whatever.  Just how much more flexibility
> than that is needed?

I remember hearing that years ago, except it was "well you can't change
global scope behaviour for EAPIs, but just how much more flexibility
than that is needed?".

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Jim Ramsay]

[-- Attachment #1: Type: text/plain, Size: 2027 bytes --]

Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> On Tue, 24 Feb 2009 15:07:29 -0500
> Jim Ramsay <lack@gentoo.org> wrote:
> > I think
> > things are very nicely documented in PMS and the devmanual, which
> > are where all EAPI changes should be documented in the future,
> > regardless if you specify the EAPI in the file, the extension, or
> > both.
> 
> They only ended up nicely documented after people moaned a lot that
> they were having a hard time keeping track of EAPIs...

You can't possibly be suggesting that everyone will be able to keep an
ever-increasing number of feature sets in his or her mind, or that
changing from a two-level to a one-level EAPI definition will remove
the need for documentation going forward, so I'm not sure what you mean
by this.

> > Two levels really just means that any fancy tables will have to have
> > one extra row (or perhaps a series of fancy tables) and any
> > summaries will have to have an extra section added whenever a new
> > filename extension becomes necessary.
> 
> It'll mean people will carry on having to use the tables, and won't
> start remembering things as time goes on.

See comment above.  The need for documentation will only increase going
forward as new and varied EAPI definitions are created.

> > If I understand the '.eapi3.eb' to which you make passing reference,
> > this is just a fancy hand-wavy way to say "Look, the true .eb
> > extension won't ever change, just the .eapi3 part which isn't
> > technically the extension..." which isn't a compromise at all - It's
> > an attempt to (cleverly?) obfuscate where in the filename the EAPI
> > is stored.
> 
> Yup. And yet there're people who are perfectly happy with .eapi3.eb
> who hate GLEP 55. That should tell you all you need to know about
> what's going on here...

Seriously?  That's scary.  But hey, if that's actually going to get
more people behind this, let's do it instead.

-- 
Jim Ramsay
Gentoo Developer (rox/fluxbox/gkrellm/vim)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Robert Bridge]

Ciaran McCreesh wrote:
> On Tue, 24 Feb 2009 09:36:29 -0700
> Joe Peterson <lavajoe@gentoo.org> wrote:
>> 2) it makes sense to have these in the filename, but not
>> internal meta-data
> 
> For those of us who understand the process, it makes sense to have EAPI
> in the filename too.

Which seems to be an enlightened few who... How did we manage before you
graced us with your presence?!

*humbly prostrates myself before this paragon of enlightenment*

Robbie

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Tue, 24 Feb 2009 15:37:36 -0500
Jim Ramsay <lack@gentoo.org> wrote:
> > They only ended up nicely documented after people moaned a lot that
> > they were having a hard time keeping track of EAPIs...
> 
> You can't possibly be suggesting that everyone will be able to keep an
> ever-increasing number of feature sets in his or her mind, or that
> changing from a two-level to a one-level EAPI definition will remove
> the need for documentation going forward, so I'm not sure what you
> mean by this.

That's exactly what I mean. Developers can probably just about keep up
with the two or three EAPIs they'll ever be working with on a regular
basis, but they probably can't keep up with that if you double it.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Jim Ramsay]

[-- Attachment #1: Type: text/plain, Size: 1492 bytes --]

Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> On Tue, 24 Feb 2009 15:37:36 -0500
> Jim Ramsay <lack@gentoo.org> wrote:
> > > They only ended up nicely documented after people moaned a lot
> > > that they were having a hard time keeping track of EAPIs...
> > 
> > You can't possibly be suggesting that everyone will be able to keep
> > an ever-increasing number of feature sets in his or her mind, or
> > that changing from a two-level to a one-level EAPI definition will
> > remove the need for documentation going forward, so I'm not sure
> > what you mean by this.
> 
> That's exactly what I mean. Developers can probably just about keep up
> with the two or three EAPIs they'll ever be working with on a regular
> basis, but they probably can't keep up with that if you double it.

Well, if you're assuming only two or three EAPIs in 'mental cache' at
any one time under glep-55, I'm not sure how this changes wrt. a
two-level system.  A two-level system doesn't change the number of
EAPIs in the tree or available to developers, it just changes how they
are named.

Regardless, this does not remove the need for documentation.  All
the EAPIs should be documented in both the PMS and the devmanual.  This
makes it possible for new developers to learn about the current
features available, and also helps existing devs who may need to
recover from 'mental page faults' from time to time.

-- 
Jim Ramsay
Gentoo Developer (rox/fluxbox/gkrellm/vim)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Roy Bamford]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2009.02.24 16:48, Ciaran McCreesh wrote:
[snip]
> 
> PN and PV are metadata, same as EAPI.
> 
[snip]
> -- 
> Ciaran McCreesh
> 

So we made a mistake with PN and PV and may compound it with EAPI.
How long before we *must* have other pieces of information in the 
filename?

When will the filename get so long as to become unwhieldy ?
Lets fix it properly now since it will be much more painful to put an 
extendable solution in place later.

It reminds me of other hacks in the history of the PC which we would do 
well not to repeat.
e.g. the MSDOS Partition Table, the Extended Partition, the High Memory 
Area.  

- -- 
Regards,

Roy Bamford
(NeddySeagoon) a member of
gentoo-ops
forum-mods
treecleaners
trustees
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (GNU/Linux)

iEYEARECAAYFAkmkZAsACgkQTE4/y7nJvatrUwCgpEKpHAF0hdbkZIJavjwXtABT
eXAAoPlWbQ0B4+F3YFPjrjCXHjuwS2m1
=r7kn
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

On Tue, 24 Feb 2009 21:17:57 +0000
Roy Bamford <neddyseagoon@gentoo.org> wrote:
> > PN and PV are metadata, same as EAPI.
>
> So we made a mistake with PN and PV and may compound it with EAPI.
> How long before we *must* have other pieces of information in the 
> filename?

Uh, never.

> When will the filename get so long as to become unwhieldy ?

Uh, never.

> Lets fix it properly now since it will be much more painful to put an 
> extendable solution in place later.

55 is the fix.

> It reminds me of other hacks in the history of the PC which we would
> do well not to repeat.
> e.g. the MSDOS Partition Table, the Extended Partition, the High
> Memory Area.  

Except that once we have EAPI in the file extension, we can change
anything we want in arbitrary ways without having to worry about
backwards compatibility, so we won't need silly hacks.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Robert Bridge]

Ciaran McCreesh wrote:
> On Tue, 24 Feb 2009 21:17:57 +0000
> Except that once we have EAPI in the file extension, we can change
> anything we want in arbitrary ways without having to worry about
> backwards compatibility, so we won't need silly hacks.

Like the file name structure?

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Tue, 24 Feb 2009 15:17:01 -0500
> Richard Freeman <rich0@gentoo.org> wrote:
>> Why?  Just parse the EAPI out of the file before you interpret the 
>> version and name from the filename.  Indeed - you could have a future 
>> EAPI remove the name and version from the filename entirely.  If a 
>> package manager doesn't understand the EAPI in a file it shouldn't do 
>> anything at all with it.
> 
> Then you get into the mess of deciding what is or is not an ebuild...
> Currently it's well defined; if you start making the package manager
> look inside files things get very confusing...

an ebuild is something ending with .ebuild ...

> It means opening a file that would otherwise not be opened at all. And
> if the EAPI is in the file, you have to fish inside that file to pull
> it out before you can work out whether the cache entry that might
> contain the EAPI already is valid.

Keeping in mind that:
- if the cache is present you won't do it (so normal users aren't touched)
- you just need a way to upgrade portage and nothing else.

You:
- have to open them on regen, no matter what (you are adding it to portage)
- the cache entry has already the eapi value so there it is.

>>> ..and it means we can't make arbitrary format changes.
>> Well, you would need to preserve the EAPI in the header, but other
>> than that you could actually turn an ebuild into an otherwise
>> completely binary file, or whatever.  Just how much more flexibility
>> than that is needed?
> 
> I remember hearing that years ago, except it was "well you can't change
> global scope behaviour for EAPIs, but just how much more flexibility
> than that is needed?".

Given that the fixed header gives you ALL the flexibility. You may give 
provision to consider the next bytes as any kind of serialization...

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On Tue, 24 Feb 2009 22:46:17 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> > It means opening a file that would otherwise not be opened at all.
> > And if the EAPI is in the file, you have to fish inside that file
> > to pull it out before you can work out whether the cache entry that
> > might contain the EAPI already is valid.
> 
> Keeping in mind that:
> - if the cache is present you won't do it (so normal users aren't
> touched)
> - you just need a way to upgrade portage and nothing else.

Not true. You don't know whether the cache is valid until you know what
the EAPI is.

> You:
> - have to open them on regen, no matter what (you are adding it to
> portage)
> - the cache entry has already the eapi value so there it is.

Can't use the cache until you know what the EAPI is.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Roy Bamford]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2009.02.24 21:23, Ciaran McCreesh wrote:
> On Tue, 24 Feb 2009 21:17:57 +0000
> Roy Bamford <neddyseagoon@gentoo.org> wrote:
> > > PN and PV are metadata, same as EAPI.
> >
> > So we made a mistake with PN and PV and may compound it with EAPI.
> > How long before we *must* have other pieces of information in the 
> > filename?
> 
> Uh, never.
> 
> > When will the filename get so long as to become unwhieldy ?
> 
> Uh, never.
> 
[snip]

> Except that once we have EAPI in the file extension, we can change
> anything we want in arbitrary ways without having to worry about
> backwards compatibility, so we won't need silly hacks.
> 
> -- 
> Ciaran McCreesh
> 
Ciaran,

Your response amounts to empty assertions.  Never is a long time.
It reminds me of a similar assertion that "640kB [of RAM] will be 
enough for anyone".

- -- 
Regards,

Roy Bamford
(NeddySeagoon) a member of
gentoo-ops
forum-mods
treecleaners
trustees
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (GNU/Linux)

iEYEARECAAYFAkmkcKkACgkQTE4/y7nJvaujMwCdES699P2BXc6VBRKfG4S9As4o
CakAnAp8tsStf1NfKp1AEsGAQC8yxuoE
=zCeb
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

On Tue, 24 Feb 2009 22:11:47 +0000
Roy Bamford <neddyseagoon@gentoo.org> wrote:
> > Except that once we have EAPI in the file extension, we can change
> > anything we want in arbitrary ways without having to worry about
> > backwards compatibility, so we won't need silly hacks.
> 
> Your response amounts to empty assertions.  Never is a long time.
> It reminds me of a similar assertion that "640kB [of RAM] will be 
> enough for anyone".

Er, no. Think about it. If EAPI is in file extension, the only concern
for backwards compatibility is not reusing an existing valid extension.
All we have to do is use a new EAPI value, and then we can change
whatever we like because non-supporting package managers will ignore it.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ciaran McCreesh wrote:
> Not true. You don't know whether the cache is valid until you know what
> the EAPI is.

If you are on the user scenario the cache is valid.
If the eapi changes the cache meaning you can always put the new cache 
in another place older portage won't look into.

>> You:
>> - have to open them on regen, no matter what (you are adding it to
>> portage)
>> - the cache entry has already the eapi value so there it is.
> 
> Can't use the cache until you know what the EAPI is.

The current cache holds all the current portage needs to know what to 
ignore, providing the cache in such format will make portage ignore any 
future change.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ryan Hill wrote:
> some other random ideas i've seen tossed around:
> - #!/bin/env eapi-parser
> - split EAPI into EAPI and some separate counter which would only be 
> incremented on uncompatible changes to the ebuild format
> - change .ebuild to .eb
> - waffles (sorry, lunchtime)

Yummy!

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On Tue, 24 Feb 2009 23:48:27 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > Not true. You don't know whether the cache is valid until you know
> > what the EAPI is.
> 
> If you are on the user scenario the cache is valid.

Uh. Wrong.

> > Can't use the cache until you know what the EAPI is.
> 
> The current cache holds all the current portage needs to know what to 
> ignore, providing the cache in such format will make portage ignore
> any future change.

Uh. Wrong.

The information used to validate a cache entry is only usable if you
know the behaviour of 'inherit' that was used to create the entry.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 267 bytes --]

Also one point worth nothing here is that migrating from EAPI in the
file name to having it in a special place in the file can be scripted so
the change should be quite easy to do at a later point in time for the
main repository if wanted.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Tue, 24 Feb 2009 18:16:54 +0100
> Luca Barbato <lu_zero@gentoo.org> wrote:
>>> You're doubling the number of files that have to be read for an
>>> operation that's almost purely i/o bound. On top of that, you're
>>> introducing a whole bunch of disk seeks in what's otherwise a nice
>>> linear operation.
>> I see words, not numbers.
> 
> Number: double. That's a '2 times'.

given that the simplest thing is hacking ebuild.sh and extract eapi with 
a simple C program (you can use pcre or ragel if you want) exactly 
before the ebuild source:

Index: bin/ebuild.sh
===================================================================
--- bin/ebuild.sh	(revision 12704)
+++ bin/ebuild.sh	(working copy)
@@ -1848,6 +1848,7 @@
  	# eclasses, they need to be unset before this process of
  	# interaction begins.
  	unset DEPEND RDEPEND PDEPEND IUSE
+	EAPI=$(eapitool "${EBUILD}")
  	source "${EBUILD}" || die "error sourcing ebuild"

  	if [ "${EBUILD_PHASE}" != "depend" ] ; then

I think your numbers are a bit pessimistic, this is when you get EAPI in 
portage, post source ${EBUILD}, opening the file before source would 
just put in the cache one line earlier.

> You don't know whether the cache is valid until you know the EAPI. It
> only works currently because EAPIs don't change inherit behaviour.

There were already discussions about switching cache format, if we want 
to change the inherit behaviour we could just switch at the same time 
the cache format and leave dummy entry for compatibility with legacy 
portage.

> So you have patches for Portage? Please show them.

Up there what's enough to check the viability for the solution.

the bash subst I wrote before could be used instead of the call to get 
the eapi in extension behaviour.

>> unknown isn't unsupported?
> 
> Huh? Please explain what you mean.

mv cat/pkg-version.ebuild cat/pkg-version_foo.ebuild

emerge -vp pkg

portage will warn about not knowing pkg-version_foo and will ignore it.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Tiziano Müller]

[-- Attachment #1: Type: text/plain, Size: 1506 bytes --]

Am Dienstag, den 24.02.2009, 22:58 +0000 schrieb Ciaran McCreesh:
> On Tue, 24 Feb 2009 23:48:27 +0100
> Luca Barbato <lu_zero@gentoo.org> wrote:
> > Ciaran McCreesh wrote:
> > > Not true. You don't know whether the cache is valid until you know
> > > what the EAPI is.
> > 
> > If you are on the user scenario the cache is valid.
> 
> Uh. Wrong.
> 
> > > Can't use the cache until you know what the EAPI is.
> > 
> > The current cache holds all the current portage needs to know what to 
> > ignore, providing the cache in such format will make portage ignore
> > any future change.
> 
> Uh. Wrong.
> 
> The information used to validate a cache entry is only usable if you
> know the behaviour of 'inherit' that was used to create the entry.
> 
Well, you could theoretical consider everything in the cache valid
within the current scope, find the eapi within the cache or the ebuild
and then reconsider things.
But the problem with this approach (besides performance, etc.) is that
it is not possible to make a pm robust enough to not fail completely
when parsing the cache entry.

The point is: Since the cache format is part of the eapi (since we store
eapi-dependant information in there), the eapi must be known before
parsing the cache data.

Would it be possible to change the cache-format with with G55?
Meaning: Have the current cache-format for the current *.ebuild and
another for *.ebuild-N (where I mean by cache-format the contents of the
cache-files)?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 2035 bytes --]

On Tue, 24 Feb 2009 19:37:11 +0000
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Tue, 24 Feb 2009 20:28:43 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
> > On Tue, 24 Feb 2009 18:24:16 +0000
> > Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:
> > > On Tue, 24 Feb 2009 18:16:54 +0100
> > > Luca Barbato <lu_zero@gentoo.org> wrote:
> > > > > You're doubling the number of files that have to be read for
> > > > > an operation that's almost purely i/o bound. On top of that,
> > > > > you're introducing a whole bunch of disk seeks in what's
> > > > > otherwise a nice linear operation.
> > > > 
> > > > I see words, not numbers.
> > > 
> > > Number: double. That's a '2 times'.
> > 
> > That only means you're increasing the constant factor in the
> > complexity of the thing... which may very well be completely
> > negligible unless someone provides real benchmarks.
> 
> In the most common case where metadata is valid, around half of the
> time it takes Paludis to work out a resolution set is spent grabbing
> metadata for ebuilds.

That sounds like an implementation detail that you could solve by using
something else than a flat file database for metadata if open()/read()
calls are the slow part.

> > I would be very surprised if that "2 times" factor happens to be
> > true, because finding a string in a file is an order of magnitude
> > simpler than sourcing said file with bash. Moreover this doesn't
> > take into account disk and os cache.
> 
> No no no. *Opening* the file is the slow part, not searching. The file
> wouldn't otherwise be opened at all.

Thus the only drawback is when you open a file, see there that you can't
handle the eapi, then close it and open an older one. So you're doing
useless things only in that case which in practice will have a ratio
far lower than 2. Moreover Luca's benchs show that searching for file
names is a negligible factor faster than grepping them while you're
just stating that it isn't.

Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

On Wed, 25 Feb 2009 07:34:41 +0100
Tiziano Müller <dev-zero@gentoo.org> wrote:
> Well, you could theoretical consider everything in the cache valid
> within the current scope, find the eapi within the cache or the ebuild
> and then reconsider things.

You can't even do that, because new EAPIs might change how cache
entries are validated. An ID might have been moved to a new EAPI using
some new EAPI thing (such as changing how inherit works) that doesn't
end up altering mtimes in a way that can be caught.

> Would it be possible to change the cache-format with with G55?
> Meaning: Have the current cache-format for the current *.ebuild and
> another for *.ebuild-N (where I mean by cache-format the contents of
> the cache-files)?

If you have GLEP 55 you don't need it.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

On Wed, 25 Feb 2009 09:33:44 +0100
Alexis Ballier <aballier@gentoo.org> wrote:
> That sounds like an implementation detail that you could solve by
> using something else than a flat file database for metadata if
> open()/read() calls are the slow part.

Metadata's shipped with the tree. It's a PMS detail.

If we didn't care about package manager performance, we wouldn't be
shipping metadata with the tree at all...

> > > I would be very surprised if that "2 times" factor happens to be
> > > true, because finding a string in a file is an order of magnitude
> > > simpler than sourcing said file with bash. Moreover this doesn't
> > > take into account disk and os cache.
> > 
> > No no no. *Opening* the file is the slow part, not searching. The
> > file wouldn't otherwise be opened at all.
> 
> Thus the only drawback is when you open a file, see there that you
> can't handle the eapi, then close it and open an older one.

Uh. No. The drawback is that you're opening around ten thousand files
that would otherwise not be opened. That's a huge cost.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

On Wed, 25 Feb 2009 04:04:46 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> given that the simplest thing is hacking ebuild.sh and extract eapi
> with a simple C program (you can use pcre or ragel if you want)
> exactly before the ebuild source:

That you're bringing ebuild.sh into this shows you still haven't worked
out how the process works. There is no need to use ebuild.sh (which is
a very good thing, because launching bash is slooooooooooooow) when
there's valid metadata.

> > So you have patches for Portage? Please show them.
> 
> Up there what's enough to check the viability for the solution.

No, it's completely wrong. ebuild.sh has nothing to do with this.

> >> unknown isn't unsupported?
> > 
> > Huh? Please explain what you mean.
> 
> mv cat/pkg-version.ebuild cat/pkg-version_foo.ebuild
> 
> emerge -vp pkg
> 
> portage will warn about not knowing pkg-version_foo and will ignore
> it.

Yes, it will warn noisily. This is unacceptable, since stable users
will have months and months of noise when new rules come along.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Wed, 25 Feb 2009 04:04:46 +0100
> Luca Barbato <lu_zero@gentoo.org> wrote:
>> given that the simplest thing is hacking ebuild.sh and extract eapi
>> with a simple C program (you can use pcre or ragel if you want)
>> exactly before the ebuild source:
> 
> That you're bringing ebuild.sh into this shows you still haven't worked
> out how the process works. There is no need to use ebuild.sh (which is
> a very good thing, because launching bash is slooooooooooooow) when
> there's valid metadata.

it there is a valid metadata you have everything there already and you 
don't have to parse the ebuild so...

>>> So you have patches for Portage? Please show them.
>> Up there what's enough to check the viability for the solution.
> 
> No, it's completely wrong. ebuild.sh has nothing to do with this.

Is there another place in portage that extracts the EAPI value?

> Yes, it will warn noisily. This is unacceptable, since stable users
> will have months and months of noise when new rules come along.

"unacceptable"...

as in "it's ugly to see"...


lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Thomas Anderson]

On Wed, Feb 25, 2009 at 04:56:04PM +0100, Luca Barbato wrote:
>> Yes, it will warn noisily. This is unacceptable, since stable users
>> will have months and months of noise when new rules come along.
>
> "unacceptable"...
>
> as in "it's ugly to see"...
>

No, it's unacceptable because stable users do not need that kind of
stuff thrown at them. Stable users use stable because they want a very
predictable workflow. Noisy errors that shouldn't affect them(they are
in the stable branch) *is* unacceptable, and not just because it's ugly,
though that's certainly part of it.

-- 
---------
Thomas Anderson
Gentoo Developer
/////////
Areas of responsibility:
AMD64, Secretary to the Gentoo Council
---------

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

On Wed, 25 Feb 2009 16:56:04 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> > That you're bringing ebuild.sh into this shows you still haven't
> > worked out how the process works. There is no need to use ebuild.sh
> > (which is a very good thing, because launching bash is
> > slooooooooooooow) when there's valid metadata.
> 
> it there is a valid metadata you have everything there already and
> you don't have to parse the ebuild so...

Except you can't tell whether the metadata is valid unless you already
know the EAPI. As you have already been told.

> >>> So you have patches for Portage? Please show them.
> >> Up there what's enough to check the viability for the solution.
> > 
> > No, it's completely wrong. ebuild.sh has nothing to do with this.
> 
> Is there another place in portage that extracts the EAPI value?

Yup. The part where it reads the metadata cache.

> > Yes, it will warn noisily. This is unacceptable, since stable users
> > will have months and months of noise when new rules come along.
> 
> "unacceptable"...
> 
> as in "it's ugly to see"...

No, as in it'll result in zillions of users wondering what's going on
and why their screen is getting spammed, and zillions of bug reports and
forum posts.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Wed, 25 Feb 2009 09:33:44 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
>> That sounds like an implementation detail that you could solve by
>> using something else than a flat file database for metadata if
>> open()/read() calls are the slow part.
> 
> Metadata's shipped with the tree. It's a PMS detail.
> 
> If we didn't care about package manager performance, we wouldn't be
> shipping metadata with the tree at all...
> 
>>>> I would be very surprised if that "2 times" factor happens to be
>>>> true, because finding a string in a file is an order of magnitude
>>>> simpler than sourcing said file with bash. Moreover this doesn't
>>>> take into account disk and os cache.
>>> No no no. *Opening* the file is the slow part, not searching. The
>>> file wouldn't otherwise be opened at all.
>> Thus the only drawback is when you open a file, see there that you
>> can't handle the eapi, then close it and open an older one.
> 
> Uh. No. The drawback is that you're opening around ten thousand files
> that would otherwise not be opened. That's a huge cost.
> 

Huge cost...

emerge -uDp world (cold os cache)

real	1m10.353s
user	0m17.077s
sys	0m0.440s

with eapitool getting the eapi before sourcing.

real	1m10.636s
user	0m16.941s
sys	0m0.368s

cold cache, no metadata available

real	6m23.106s
user	3m32.141s
sys	1m50.855s

with eapitool

real	6m26.564s
user	3m31.853s
sys	1m50.511s


I'd rather see more people backing their ideas with numbers...

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Luca Barbato]

Thomas Anderson wrote:
> On Wed, Feb 25, 2009 at 04:56:04PM +0100, Luca Barbato wrote:
>>> Yes, it will warn noisily. This is unacceptable, since stable users
>>> will have months and months of noise when new rules come along.
>> "unacceptable"...
>>
>> as in "it's ugly to see"...
>>
> 
> No, it's unacceptable because stable users do not need that kind of
> stuff thrown at them. Stable users use stable because they want a very
> predictable workflow. Noisy errors that shouldn't affect them(they are
> in the stable branch) *is* unacceptable, and not just because it's ugly,
> though that's certainly part of it.

You have quite a good point here. It's purely cosmetical, but since it's 
an unexpected behavior and it's annoying and could alienate our users 
it's better to avoid it with all our strength.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 314 bytes --]

On Wed, 25 Feb 2009 17:17:29 +0100
Luca Barbato <lu_zero@gentoo.org> wrote:
> I'd rather see more people backing their ideas with numbers...

I already told you your numbers are nonsense.

Of course opening the file when you've already opened it isn't going to
make any difference.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Peter Volkov]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]

В Втр, 24/02/2009 в 16:14 +0200, Serkan Kaba пишет:
> lucene-contrib eclass in java-experimental [1] sets EAPI to 1 to use
> slot deps. And I think that's a valid usage.
> 
> 1:
> http://overlays.gentoo.org/proj/java/browser/java-experimental/eclass/lucene-contrib.eclass

It's better (the only way...) to die in case an ebuild sets lower EAPI,
like kde4-functions.eclass does:

case ${EAPI} in
    2) : ;;
    *) die "No way! EAPI older than 2 is not supported." ;;
esac

-- 
Peter.

[-- Attachment #2: Эта часть сообщения подписана цифровой подписью --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: Issues regarding glep-55 (Was: [gentoo-council] Re: Preliminary Meeting-Topics for 12 February 2009)

[Matthias Schwarzott]

On Samstag, 28. Februar 2009, Peter Volkov wrote:
> В Втр, 24/02/2009 в 16:14 +0200, Serkan Kaba пишет:
> > lucene-contrib eclass in java-experimental [1] sets EAPI to 1 to use
> > slot deps. And I think that's a valid usage.
> >
> > 1:
> > http://overlays.gentoo.org/proj/java/browser/java-experimental/eclass/luc
> >ene-contrib.eclass
>
> It's better (the only way...) to die in case an ebuild sets lower EAPI,
> like kde4-functions.eclass does:
>
> case ${EAPI} in
>     2) : ;;
>     *) die "No way! EAPI older than 2 is not supported." ;;
> esac

I still dislike die in global scope, why not do it like autotools.eclass?
It does:
DEPEND="INCORRECT-WANT_AUTOCONF-SETTING-IN-EBUILD"

Regards
Matthias