[gentoo-dev] GLEP 55 Version 2

[gentoo-dev] GLEP 55 Version 2

[Roy Bamford]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ladies and Gentlemen,

I've spent some time reading all of this years emails on GLEP55 and 
added a few lines to version 1.5 which is the last offical version.

The HTML version (not with the GLEP style sheet) is at 
http://xrl.us/bevrnb (my devspace)
Its not ready to go to council as it still needs additional content 
which I don't have the background to contribuite. My role in this 
version of GLEP 55 has been interpretation and editorial.

I'm looking for three elements to add.

1. A description of the steps the PM needs to extract the EAPI in 
ebuilds today, if its any more than sourcing the ebuild.

2. Benchmarks and supporting code for EAPI extraction for EAPI in the 
filename.

3. Benchmarks and supporting code for EAPI extraction for EAPI in the 
ebuild.

Benchmarks for valid and invalid cache need to be supplied so we have 
both the user and developer views of performace. 

I expect items 2 and 3 to be provided by the proponents of these two 
competing options and critically examined for validity by the 
competitors.

When the above data is available, I'll update the document as it needs 
to contain all of the evidence to allow council to reach a decision 
without reading the ml. 

Reply with corrections and factual additions with supporting evidence 
only please. The role of the GLEP is to present the evidence and make a 
reccomendation. Its councils job to make any subjective assessments 
having read the GLEP.

- -- 
Regards,

Roy Bamford
(NeddySeagoon) a member of
gentoo-ops
forum-mods
treecleaners
trustees
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)

iEYEARECAAYFAkoq7l8ACgkQTE4/y7nJvatqjQCeNe/mqS8goPZ5+H4yftn19mYJ
5i0AoLwsAmKcc6Ux4zjG3dExgRbcltwl
=edR+
-----END PGP SIGNATURE-----

[gentoo-dev] Re: GLEP 55 Version 2

[Steven J Long]

Roy Bamford wrote:
> I've spent some time reading all of this years emails on GLEP55 and
> added a few lines to version 1.5 which is the last offical version.
>
Thanks for all the hard work.

My apologies for my mistaken comment at the end of the last Council meeting.
Clearly the mangler /does/ need to know the EAPI without sourcing for future
extensibility.

I'd just like to know what the implications would be for users if we kept
the .ebuild extension, and a new PMS were rolled out stating that the
mangler were allowed to find the EAPI without sourcing (and giving the
restrictions) once portage 2.2 was stable, or the ability to handle this
backported to 2.1.6, and issued in a release?

Would it be unacceptable to have a clear upgrade path for any users who
didn't actually update portage? (Perhaps a news item so they'd be
notified as and when they ran emerge.)

It strikes me that we went through a similar situation with bash-3.17 I
think it was, and that once we're past this, there'll never be any need
to worry in the future. So, given that it's taken so long and so much
discussion, why not just do this last big bump, and not worry about
about using another extension at all?

After all, the issue would only arise once Council approved an EAPI
requiring one of the incompatible changes, and 3 has only recently come
out.

Also, you asked for proponents of either side to provide statistics as to
the time difference between the various options. It's rather hard for me
to patch paludis, nor do I have the inclination. I am not being facetious,
simply pointing out that the comparison has to be made within a single app.
Comparing an approach implemented in portage vs one in paludis is comparing
apples and oranges.

Also, Patrick brought up cache improvements (like not having a single cache
file per ebuild, but rather per package. This could have the EAPI and
versions first, followed by metadata per version.) I feel we should bear in
mind that there are other areas where we can improve things, many of which
we have not even thought of, or at least discussed.

With respect to time gains in interactivity, much has been made of the
speed difference between portage and paludis. I have yet to see any
comparative numbers between pkgcore and paludis in this regard. (If we are
going to compare manglers and not algorithms.)

I think we are agreed now that an EAPI which can be extracted before source
does fulfil all the criteria (as others have said, this is actually what the
GLEP is about; ascertaining the EAPI without needing to source.) If not,
could someone please explain why not.

Regards,
Steve.
--
#friendly-coders -- We're friendly but we're not /that/ friendly ;-)

[gentoo-dev] Re: GLEP 55 Version 2

[Ulrich Mueller]

>>>>> On Sun, 07 Jun 2009, Steven J Long wrote:

> I'd just like to know what the implications would be for users if we
> kept the .ebuild extension, and a new PMS were rolled out stating
> that the mangler were allowed to find the EAPI without sourcing (and
> giving the restrictions) once portage 2.2 was stable, or the ability
> to handle this backported to 2.1.6, and issued in a release?

Even if we do only a one-time change of the file extension, can we
ever get rid of the old extension? Or are we then stuck with two
extensions in the tree until the end of time?

Let's assume for the moment that we change from ".ebuild" to ".eb".
Then we obviously cannot change all ebuilds in the tree to ".eb",
otherwise old Portage versions would see an empty tree and there would
be no upgrade path.

Or am I missing something?

Ulrich

Re: [gentoo-dev] Re: GLEP 55 Version 2

[Richard Freeman]

Ulrich Mueller wrote:
> Let's assume for the moment that we change from ".ebuild" to ".eb".
> Then we obviously cannot change all ebuilds in the tree to ".eb",
> otherwise old Portage versions would see an empty tree and there would
> be no upgrade path.
> 
> Or am I missing something?
> 

That is a good point.  Although things would probably break more 
gracefully than having old portage versions attempting to parse new 
ebuilds (maybe, maybe not).

That actually makes me wonder - if we didn't change the extension at all 
could we somehow poison the portage tree so that old versions of portage 
would abort before doing anything dumb with a meaningful error message?

As far as an upgrade path goes - we could provide a one-time tarball 
that will update portage (and its essential dependencies) to a version 
that can get users out of this bind.  If a user has a system THAT old 
then they might just want to extract a stage1 tarball (manually - 
without overwriting /etc without care) and go from there.

I'm not sure that gentoo generally supports graceful upgrades from very 
ancient systems to modern ones without keeping up to date.  Other 
distros can do it since they do ~annual releases and users could just 
apply those sequentially.  For portage we don't keep around all the 
files needed to do a sequential upgrade like this - if a user were to 
try to upgrade to a 3-year-old version of some package most likely it 
wouldn't be mirrored and upstream might not have it either.

We obviously need to give some thought to not breaking old versions of 
portage, but given that portage will be only one of many problems if a 
user doesn't do an emerge -u world for 5 years I'm not sure we need a 
bulletproof solution...

Re: [gentoo-dev] Re: GLEP 55 Version 2

[Patrick Lauer]

On Sunday 07 June 2009 11:34:12 Ulrich Mueller wrote:
> >>>>> On Sun, 07 Jun 2009, Steven J Long wrote:
> >
> > I'd just like to know what the implications would be for users if we
> > kept the .ebuild extension, and a new PMS were rolled out stating
> > that the mangler were allowed to find the EAPI without sourcing (and
> > giving the restrictions) once portage 2.2 was stable, or the ability
> > to handle this backported to 2.1.6, and issued in a release?
>
> Even if we do only a one-time change of the file extension, can we
> ever get rid of the old extension? Or are we then stuck with two
> extensions in the tree until the end of time?
>
> Let's assume for the moment that we change from ".ebuild" to ".eb".
> Then we obviously cannot change all ebuilds in the tree to ".eb",
> otherwise old Portage versions would see an empty tree and there would
> be no upgrade path.
>
> Or am I missing something?

I come to the same conclusion.

Which means that the easiest way to get things migrated will be a one-time 
change of the _sync_ location so that users have a chance to get to an 
upgrade-ready state on their system, then change sync location, then upgrade 
to the current state.

In which case we actually do not have to change the file name anymore ...

Of course things aren't always that easy, but if you add a "generation file" 
somewhere in the rsync checkout it is easy to see if your current rsync 
location is "old and deprecated" or "new and improved". And if you really 
absolutely have to do that you can change the sync location on every 
disruptive change, but (imo) that should be avoided. Less disruptive changes 
is better :)

hth,

Patrick

Re: [gentoo-dev] Re: GLEP 55 Version 2

[Federico Ferri]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ulrich Mueller wrote:
>>>>>> On Sun, 07 Jun 2009, Steven J Long wrote:
>
>> I'd just like to know what the implications would be for users if we
>> kept the .ebuild extension, and a new PMS were rolled out stating
>> that the mangler were allowed to find the EAPI without sourcing (and
>> giving the restrictions) once portage 2.2 was stable, or the ability
>> to handle this backported to 2.1.6, and issued in a release?
>
> Even if we do only a one-time change of the file extension, can we
> ever get rid of the old extension?
unfortunately, no.
>  Or are we then stuck with two
> extensions in the tree until the end of time?
> Let's assume for the moment that we change from ".ebuild" to ".eb".
better put this new ebuild type in a new tree; such a massive change
to the tree its not recommended.
> Then we obviously cannot change all ebuilds in the tree to ".eb",
> otherwise old Portage versions would see an empty tree and there would
> be no upgrade path.
leaving actual ".ebuild"s as they are now (good healthy state :)) and
making new development of ".eb" ebuilds happen in a new tree (I said
new tree, but it could be any way that hides those new ebuild to OLD
package managers) would help.

only newer versions of package managers are required to support this,
that is they will look for .eb (in new or current tree, not sure
what's best) and then for .ebuilds, and ideally this should be
transparent to old package managers and allow an upgrade path.

- --
mescalinum@g.o
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkortVIACgkQV/B5axfzrPsTiACeJCJb3F8Up/+CjHIwC3Slhn/6
yZgAoLcJgNn2d3W/JeZPkK85arUPW9vV
=fR4T
-----END PGP SIGNATURE-----

[gentoo-dev] Re: GLEP 55 Version 2

[Duncan]

Richard Freeman <rich0@gentoo.org> posted 4A2BAAA9.4030503@gentoo.org,
excerpted below, on  Sun, 07 Jun 2009 07:55:21 -0400:

> As far as an upgrade path goes - we could provide a one-time tarball
> that will update portage (and its essential dependencies) to a version
> that can get users out of this bind.  If a user has a system THAT old
> then they might just want to extract a stage1 tarball (manually -
> without overwriting /etc without care) and go from there.

We've done the tarball thing a couple times before, with portage I think, 
with amd64/gcc for certain, as it was needed to get out of some sort of 
multilib and profile based bind IIRC, and with the in-tree profiles (from 
pre-cascade profiles) at least once too, IIRC.

> I'm not sure that gentoo generally supports graceful upgrades from very
> ancient systems to modern ones without keeping up to date.  Other
> distros can do it since they do ~annual releases and users could just
> apply those sequentially.  For portage we don't keep around all the
> files needed to do a sequential upgrade like this - if a user were to
> try to upgrade to a 3-year-old version of some package most likely it
> wouldn't be mirrored and upstream might not have it either.

AFAIK from what I've read here over the years, Gentoo tries to keep 
smooth in-tree upgrades to a year out.  Beyond that, we don't usually 
deliberately break it without some warning and a tarball or similar 
upgrade path for another six months to a year, but it's by no means 
guaranteed it'll be a smooth upgrade after a year even if we aren't 
deliberately breaking it.  Generally, beyond a year, it's recommended 
that one uses the stage tarball to get something at least operationally 
modern, and goes from there.

Simply put, Gentoo's NOT in practice a distribution for the folks who 
like to lollygag around for years between updates.  Tho we do try to 
support it up to a year out and to provide at least some form of likely 
non-routine upgrade option beyond that, it definitely works best and with 
the least trouble for those updating every month or at least once a 
quarter, with things getting progressively more difficult and troublesome 
the further out beyond that you go, simply because of lack of testing if 
nothing else.

> We obviously need to give some thought to not breaking old versions of
> portage, but given that portage will be only one of many problems if a
> user doesn't do an emerge -u world for 5 years I'm not sure we need a
> bulletproof solution...

I just realized that I'm right about at my Gentoo 5-year anniversary, 
with an original installation of 2004.1.  (I tried 2004.0 but it didn't 
work for some reason I never did figure out, but perhaps related to the 
then new NPTL, which I was trying to enable.)

I can't /imagine/ first installing it then, and coming back to it now, 
expecting anything but a full reinstall from stage tarball (assuming as 
suppose I would be if I had been that out of it, that was still even 
/using/ stage tarballs as it was then).  Imagine people wondering what 
happened to xfree86, among other things.  I mean, talk about a time-
traveler getting confused by the future!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: GLEP 55 Version 2

[Richard Freeman]

Patrick Lauer wrote:
> And if you really absolutely have to do that you can change the sync
> location on every disruptive change, but (imo) that should be
> avoided.

If mirroring and other practical concerns weren't an issue what you're 
essentially describing is just moving to a CVS/git/etc repository and 
using such a tool to do all the syncs (ie not using rsync at all).  Then 
all you need is an update page that has a list of commands like:

emerge --sync --date "2008-01-01"

emerge -u world

emerge --sync --date "2008-08-10"

Follow this xorg update doc: <link>

emerge --sync --date "2034-04-02"

emerge -u portage so that you have support for the finally-approved glep55

And so on...  :)

That actually gives you a best-of-both-worlds option: continue to use 
rsync for normal use to ease distribution, but have a cvs tree that 
anybody can read from which could be used in upgrade guides for 
out-of-date users.

Re: [gentoo-dev] Re: GLEP 55 Version 2

[Roy Bamford]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2009.06.07 10:34, Ulrich Mueller wrote:
> >>>>> On Sun, 07 Jun 2009, Steven J Long wrote:
> 
> > I'd just like to know what the implications would be for users if 
> we
> > kept the .ebuild extension, and a new PMS were rolled out stating
> > that the mangler were allowed to find the EAPI without sourcing 
> (and
> > giving the restrictions) once portage 2.2 was stable, or the 
> ability
> > to handle this backported to 2.1.6, and issued in a release?
> 
> Even if we do only a one-time change of the file extension, can we
> ever get rid of the old extension? Or are we then stuck with two
> extensions in the tree until the end of time?
> 
> Let's assume for the moment that we change from ".ebuild" to ".eb".
> Then we obviously cannot change all ebuilds in the tree to ".eb",
> otherwise old Portage versions would see an empty tree and there 
> would
> be no upgrade path.
> 
> Or am I missing something?
> 
> Ulrich

First, lets be clear that the upgrade path related problems are driven 
by the EAPI change, not the .ebuild to .eb change.  That serves only to 
allow the new EAPI to be introduced immedately and hide the change from 
older package managers 

The upgrade path issue remains even if we do the usual Gentoo introduce 
new features to the package manager then wait a year or so before they 
are deployed. Without the .ebuild to .eb change. late upgraders will 
get the error messages in Appendix A of the GLEP when these features 
are relesed. 

To keep an upgrade path, package managers and their dependencies need 
to use EAPI 0, 1 or 2. (EAPI 3 is not yet out) How far back should the 
upgrade path extend?

As you suggest, even with .ebuild to .eb change.its not a clean break 
with the past but would happen over time, with ebuilds for new package 
versions being migrated to the new format. For example we would 
have 
foo-2.1-r1.ebuild
foo-3.0.eb
portage-2.3.ebuild
portage-3.0.eb

Portage-2.3 will understand the later EAPI but will itself, use only 
EAPI, 0, 1 or 2.
 
With the .ebuild to .eb change. users who do not upgrade portage will 
not see newer versions of foo. Without it, they will get the error 
messages in Appendix A of the GLEP. This will have the side effect of 
driving the adoption of the newer package managers.

It is not suffcient to have portage-2.3.ebuild as understanding later 
EAPI files. All of its dependencies need to keep to EAPI 0, 1 or 2. 
These must be the last packages to migrate to later EAPIs. 

Thats just portage. The same reasoning applies to any other package 
manager and there are at least three. This begs the question of how 
friendly do we want to be to derivitive distros that use our tree but 
their own package manager?

Upgrade path issues need to be addressed in the GLEP. I will add 
something like the above in the next version.

- -- 
Regards,

Roy Bamford
(NeddySeagoon) a member of
gentoo-ops
forum-mods
treecleaners
trustees



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)

iEYEARECAAYFAkoryrUACgkQTE4/y7nJvashtQCgkO+feHG2BBGLOTObrwq72iOx
nI4AoNZ67Mhq4yEKYTzfBOPmu98Py9Mc
=GB62
-----END PGP SIGNATURE-----

Re: [gentoo-dev] GLEP 55 Version 2

[Michael Haubenwallner]

On Sat, 2009-06-06 at 23:31 +0100, Roy Bamford wrote:

> I've spent some time reading all of this years emails on GLEP55 and 
> added a few lines to version 1.5 which is the last offical version.

Is there a specific reason not to include the "inherit eapi"[1][2]
thing?

IMHO this would fit best in "Abstract Solution (Part 1)" somehow like:

        -There are two potential solutions to this part of the problem,
        +There are three potential solutions to this part of the
        problem,
                
            * wait for old package managers to fall into disuse
            * 'blind' old package managers to ebuilds using later
        constructs
        +   * use "inherit" to have old package managers almost silently
        +     ignore unsupported ebuilds

[1] http://archives.gentoo.org/gentoo-dev/msg_b2656aca1d124658b3df73d3d6804b7e.xml
[2] http://archives.gentoo.org/gentoo-dev/msg_348f84690f03e597fb14d6602337c45f.xml

Thanks!

/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

[gentoo-dev] Re: Re: GLEP 55 Version 2

[Steven J Long]

Roy Bamford wrote:

> On 2009.06.07 10:34, Ulrich Mueller wrote:
>> >>>>> On Sun, 07 Jun 2009, Steven J Long wrote:
>> 
>> > I'd just like to know what the implications would be for users if
>> we
>> > kept the .ebuild extension, and a new PMS were rolled out stating
>> > that the mangler were allowed to find the EAPI without sourcing
>> (and
>> > giving the restrictions) once portage 2.2 was stable, or the
>> ability
>> > to handle this backported to 2.1.6, and issued in a release?
>> 
>> Even if we do only a one-time change of the file extension, can we
>> ever get rid of the old extension? Or are we then stuck with two
>> extensions in the tree until the end of time?
>> 
>> Let's assume for the moment that we change from ".ebuild" to ".eb".
>> Then we obviously cannot change all ebuilds in the tree to ".eb",
>> otherwise old Portage versions would see an empty tree and there
>> would
>> be no upgrade path.
>> 
>> Or am I missing something?
>>
Sounds about right

> 
> First, lets be clear that the upgrade path related problems are driven
> by the EAPI change, not the .ebuild to .eb change.  That serves only to
> allow the new EAPI to be introduced immedately and hide the change from
> older package managers
>
<snip> 
> To keep an upgrade path, package managers and their dependencies need
> to use EAPI 0, 1 or 2. (EAPI 3 is not yet out) How far back should the
> upgrade path extend?
>
Well given that EAPI 3 is not out, and that bash-4 is not even stable
(and if anyone thinks we can rely on bash-4 in the next 6 months, they
didn't learn anything from bash-3 imo) this sounds like it's a fair
distance away. From discussion with Harring, EAPIs were not meant to
come out very often; iirc he said something like maybe once a year.

I concur with Duncan on a year, as you know. I appreciate you feel it
should be longer, and am happy to go with whatever the consensus is.

> As you suggest, even with .ebuild to .eb change.its not a clean break
> with the past but would happen over time, with ebuilds for new package
> versions being migrated to the new format. For example we would
> have
> foo-2.1-r1.ebuild
> foo-3.0.eb
> portage-2.3.ebuild
> portage-3.0.eb
>
yuck.

> Portage-2.3 will understand the later EAPI but will itself, use only
> EAPI, 0, 1 or 2.
>
Just to be clear: portage-2.2 and later will be fine with any EAPI, with
no change to any ebuilds, nor any new extension being needed. The
change can easily be backported to 2.1.6, tho I suspect 2.2 will be
stable fairly soon.
 
> Thats just portage. The same reasoning applies to any other package
> manager and there are at least three. This begs the question of how
> friendly do we want to be to derivitive distros that use our tree but
> their own package manager?
>
As friendly as we can be without hobbling our own development. Most of
them appear to be fairly collaborative with Gentoo in any case.

> Upgrade path issues need to be addressed in the GLEP. I will add
> something like the above in the next version.
>
Wrt transitioning, can the eapi (PMS 5.2.2) and deprecated (5.2.3) files
not be of use? I seem to recall the change from 2007.1 to 2008.0 for
example; anyone using a deprecated profile can expect a massive warning
the next time they try to do anything after sync'ing.

Thanks again for your work; I appreciate that your time is valuable.

Regards,
Steve.
--
#friendly-coders -- We're friendly but we're not /that/ friendly ;-)