[gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

Recently, Prefix changes have been committed to the gentoo-x86 tree, it
was rather ambitious on my part, where I moved stuff that we are not
maintainer of ourself.  It should have been communicated better for
these ebuilds.  This is a formal apology for springing that onto you.
This will attempt to solve the confusion and answer questions that have
been raised.

Gentoo Prefix has mainly the following characteristics:
- allow for unprivileged installations,
- in an offset-prefix called "the prefix", often referred to as EPREFIX,
  in autoconf words: ./configure --prefix=${EPREFIX}/usr

In the Prefix Portage branch, that we have in use [1], we extended and
adapted regular Portage to support above two characteristics.  I will
ignore the details of unprivileged installations in this email.

The offset-prefix changes in Portage introduce three new variables that
are available in the ebuild environment:
- EPREFIX: the configured offset-prefix
- ED: shorthand for ${D%/}${EPREFIX}/
- EROOT: ${ROOT%/}${EPREFIX}/

The offset-prefix changes require extensive changes to most eclasses,
and minor changes to many ebuilds.  This is mainly because "awareness"
of the offset-prefix has to be added to places where ebuilds manually
deal with file-system locations.  In particular:
- configure calls that specify some argument to find a component need to
  do so in the offset prefix, e.g.
  --with-libxml2="${EPREFIX}"/usr/$(get_libdir)
- almost every place where ${D} is used, the offset-prefix ${EPREFIX}
  has to be added.  Because this is lengthy, Prefix Portage provides a
  variable ${ED} that is the shortcut for ${D} plus the offset-prefix.
- the same holds for occurrences of ${ROOT}, where ${EROOT} is available
  as shortcut

Why do we need both ${ED} and ${D}?  Technically we don't, because we
can use ${EPREFIX} all the time.  However, one cannot say that ${D}
includes ${EPREFIX} for Prefix Portage, because that means the following
src_install() function no longer works correctly:

  src_install() {
    emake DESTDIR="${D}" install || die
    mv "${D}"/usr/bin/{,exuberant-}ctags || die
  }

While the "mv" does a great job if ${D} would include ${EPREFIX} here,
the make install will cause double offset to be written, since configure
recorded the ${EPREFIX} before in src_compile using the argument
--prefix="${EPREFIX}"/usr.  In a previous version of Prefix Portage, the
variable EDEST was available as workaround for this, so the example
would read like this:

  src_install() {
    emake DESTDIR="${EDEST}" install || die
    mv "${D}"/usr/bin/{,exuberant-}ctags || die
  }

Apart from that this approach got very tricky and confusing in eclasses
and ebuilds that do special mungling in their src_install, it makes it
harder to reconstruct the variable in normal Portage and hence to make
existing ebuilds forward compatible.

The lengthy forward compatible version of the example src_install
function would look like this:

  src_install() {
    emake DESTDIR="${D}" install || die
    mv "${D%/}${EPREFIX}"/usr/bin/{,exuberant-}ctags || die
  }

As mentioned before, this is pretty lengthy, and quickly becomes too
much work and unreadable when ${D} is used more.
To avoid the confusion that implicit ${EPREFIX} in ${D} in the ${EDEST}
approach brought, ED and EROOT were chosen because they are easy to
understand and easy to reconstruct outside Prefix Portage.  To help
with this, the Prefix profiles use.force the "prefix" USE-flag.
Non-Prefix profiles have this flag masked and unset in the base profile.
This USE-flag can be used to do conditional code for Prefix consumers.
In case of our ${ED} and ${EROOT} convenience variables, we can use it
to define ${ED} and ${EROOT} in case a normal Portage is used.  For our
example function:

  src_install() {
    use prefix || local ED=${D}
    emake DESTDIR="${D}" install || die
    mv "${ED}"/usr/bin/{,exuberant-}ctags || die
  }

Here, Prefix Portage (using a Prefix profile) will not set ${ED}, but
still do as intended because ${ED} is set correctly by Prefix Portage.
Normal Portage will set the convenience variable such that it does not
cause a sandbox violation due to the missing image path.

We will consult the maintainers of packages we would like to make
compatible with Gentoo Portage before adding the changes.  In the
future, you will likely see more bug reports and more requests on IRC
from us.

At this point, we have reached a critical mass where we cannot maintain
the Prefix overlay much longer with its size and usage.  We either
continue to grow on, requiring less maintenance on tree synchronisation,
or we stop the project -- an option we really don't like.
Bringing back most, and ideally all, ebuilds to gentoo-x86 significantly
improves the workload caused by synchronisation, leaving more time for
the real issues.  Examples are fixing and porting packages and getting
the Prefix Portage branch merged with regular Portage some day.  At that
point, the variables EPREFIX, ED and EROOT can become available in a
next EAPI as well.


[1] http://sources.gentoo.org/viewcvs.py/portage/main/branches/prefix/

-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Tomáš Chvátal]

Hi,
You know i am totaly supporting prefix but i have one point.
Why on earth portage simply does not detect the prefix enviroment is being run 
and then INTERNALY switch D->ED and other variables. It would be much easier 
that way to migrate all stuff in portage instead of doing this || shebang. 
Mostly when it is done by eclasses its quite cool, but when you get into 
changing lots of ebuilds its quite hard for maintaining.

Even the multilib overlay guys rather modify the portage than changing a load 
of ebuilds.

Just my 2 cents.

Tomas

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 18-10-2009 13:57:10 +0200, Tomáš Chvátal wrote:
> Hi,
> You know i am totaly supporting prefix but i have one point.
> Why on earth portage simply does not detect the prefix enviroment is being run 
> and then INTERNALY switch D->ED and other variables. It would be much easier 
> that way to migrate all stuff in portage instead of doing this || shebang. 
> Mostly when it is done by eclasses its quite cool, but when you get into 
> changing lots of ebuilds its quite hard for maintaining.
> 
> Even the multilib overlay guys rather modify the portage than changing a load 
> of ebuilds.

Of course we would like to do that, but that was rejected for EAPI=3, so
it will at least take until EAPI=4 is implemented, which is not the
forseeable future, given that EAPI=3 isn't a fact yet either.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 18-10-2009 14:31:15 +0200, Fabian Groffen wrote:
> On 18-10-2009 13:57:10 +0200, Tomáš Chvátal wrote:
> > Hi,
> > You know i am totaly supporting prefix but i have one point.
> > Why on earth portage simply does not detect the prefix enviroment is being run 
> > and then INTERNALY switch D->ED and other variables. It would be much easier 
> > that way to migrate all stuff in portage instead of doing this || shebang. 
> > Mostly when it is done by eclasses its quite cool, but when you get into 
> > changing lots of ebuilds its quite hard for maintaining.
> > 
> > Even the multilib overlay guys rather modify the portage than changing a load 
> > of ebuilds.
> 
> Of course we would like to do that, but that was rejected for EAPI=3, so
> it will at least take until EAPI=4 is implemented, which is not the
> forseeable future, given that EAPI=3 isn't a fact yet either.

I was just informed that it is also a possibility to do an EAPI bump
just for these variables, which would mean we can avoid replicating
setting ED and EROOT in ebuilds.

The suggestion was to just introduce EAPI=3 with these variables, and
making everything which is scheduled for current EAPI=3 just EAPI=4.  I
was told we could quite quickly have a Portage in the tree that would
set ED and EROOT for EAPI=3 that way.

Are there any objections to this?  If not, I'd like to put this on the
agenda for the next council meeting.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 1766 bytes --]

Fabian Groffen wrote:
> On 18-10-2009 14:31:15 +0200, Fabian Groffen wrote:
>> On 18-10-2009 13:57:10 +0200, Tomáš Chvátal wrote:
>>> Hi,
>>> You know i am totaly supporting prefix but i have one point.
>>> Why on earth portage simply does not detect the prefix enviroment is being run 
>>> and then INTERNALY switch D->ED and other variables. It would be much easier 
>>> that way to migrate all stuff in portage instead of doing this || shebang. 
>>> Mostly when it is done by eclasses its quite cool, but when you get into 
>>> changing lots of ebuilds its quite hard for maintaining.
>>>
>>> Even the multilib overlay guys rather modify the portage than changing a load 
>>> of ebuilds.
>> Of course we would like to do that, but that was rejected for EAPI=3, so
>> it will at least take until EAPI=4 is implemented, which is not the
>> forseeable future, given that EAPI=3 isn't a fact yet either.
> 
> I was just informed that it is also a possibility to do an EAPI bump
> just for these variables, which would mean we can avoid replicating
> setting ED and EROOT in ebuilds.
> 

It's possible.

> The suggestion was to just introduce EAPI=3 with these variables, and
> making everything which is scheduled for current EAPI=3 just EAPI=4.  I
> was told we could quite quickly have a Portage in the tree that would
> set ED and EROOT for EAPI=3 that way.
> 

Maybe 2+prefix is a more describing name? This would avoid changing what
EAPI 3 means.

> Are there any objections to this?  If not, I'd like to put this on the
> agenda for the next council meeting.
> 

As the council decided to add new stuff in the last meeting if zac is
starting to implement new EAPIs this could go into EAPI 3 too.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 24-10-2009 22:37:30 +0300, Petteri Räty wrote:
> > The suggestion was to just introduce EAPI=3 with these variables, and
> > making everything which is scheduled for current EAPI=3 just EAPI=4.  I
> > was told we could quite quickly have a Portage in the tree that would
> > set ED and EROOT for EAPI=3 that way.
> 
> Maybe 2+prefix is a more describing name? This would avoid changing what
> EAPI 3 means.

Naming is up to others, from my point of view.

> > Are there any objections to this?  If not, I'd like to put this on the
> > agenda for the next council meeting.
> 
> As the council decided to add new stuff in the last meeting if zac is
> starting to implement new EAPIs this could go into EAPI 3 too.

Yes, this was implicit, the next EAPI should contain the same support
too.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Ulrich Mueller]

In its November meeting [1], the council has unanimously expressed
support for this proposal [2].

However, there is need for additional discussion. From the council
meeting log I could extract the following open questions:

  1. What are the implications for non-prefix devs and users? 

  2. Should the Prefix team be allowed to do the necessary changes to
     ebuilds themselves, or should it be done by the respective
     maintainers?

  3. Are there any backwards compatibility or upgrade path issues for
     eclasses that must still accept EAPI 0 (where the new ED, EROOT,
     and EPREFIX variables are not defined)?

  4. EAPI numbering: Would this simply be added as an additional
     feature to EAPI 3? Or should we have an intermediate EAPI slot,
     e.g. 2.1 or 3 (and current EAPI 3 renamed to 4 in the latter
     case)?

  5. Who is going to write the exact specification (PMS patch) for
     this EAPI feature?

  6. (Any question that I've missed?)

Let's start the discussion now, in order to work out these details
before the next council meeting (December 7th).

Ulrich

[1] <http://www.gentoo.org/proj/en/council/meeting-logs/20091109.txt>
    (topic was discussed from 21:32 to 22:11 in the log's timezone)
[2] <http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml>

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Denis Dupeyron]

On Sun, Oct 18, 2009 at 2:11 AM, Fabian Groffen <grobian@gentoo.org> wrote:
> This is a formal apology for springing that onto you.

Thanks a lot. Not everybody can do such a thing as a public apology. I
will nonetheless ask the council to vote on the following during next
meeting:
    Ask Fabian to change his signature from:
    "Gentoo on a different level"
    To:
    "Failure in a different level"
;o)

2009/10/18 Tomáš Chvátal <scarabeus@gentoo.org>:
> Why on earth portage simply does not detect the prefix enviroment is being run
> and then INTERNALY switch D->ED and other variables.

If that means we can get away without touching ebuilds, apart from
changing their EAPI variable, then that's absolutely what we have to
do. I'd like things to be done the right way though (see below).

On Fri, Nov 13, 2009 at 4:43 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
> However, there is need for additional discussion. From the council
> meeting log I could extract the following open questions:

It would be preferable for the discussion to happen on this list
before the meeting or we'll end up postponing again due to having more
questions coming up at that time.

>  2. Should the Prefix team be allowed to do the necessary changes to
>     ebuilds themselves, or should it be done by the respective
>     maintainers?

I think here it's obvious that anybody who is an ebuild dev and sees
anything to fix (prefix or else) is encouraged to go ahead and do it,
as we've always done. The recommendation is and will always be to talk
to the current maintainers out of politeness and to be extra careful
(i.e. usually letting the maintainers do it) in case of
system/tricky/exotic package. We don't give full cvs access to the
whole tree to all ebuild devs for nothing.

>  4. EAPI numbering: Would this simply be added as an additional
>     feature to EAPI 3? Or should we have an intermediate EAPI slot,
>     e.g. 2.1 or 3 (and current EAPI 3 renamed to 4 in the latter
>     case)?

Here I'd add to the choices: why not release an intermediate EAPI with
the prefix stuff and whatever that has already been done for EAPI3?
The exact name of a potential intermediate EAPI is a non-problem.
However I would prefer if it were a number like 2.1 or 2.5 or even 3,
because although we currently treat the EAPI variable as a simple
string we may change our mind later and find it handy someday to use
operators on them such as >=2.1.

>  5. Who is going to write the exact specification (PMS patch) for
>     this EAPI feature?

I thought I asked Fabian to work on that at the end of the meeting. In
case I didn't then consider this as me officially asking him if he can
take care of it. Fabian is this OK with you?

Also I think it would be nice if somebody took care of a portage
patch, since I hear it is rather simple. Fabian again? Or Zac? Any
other volunteers?

I would prefer to have all the pieces in places before the next
meeting so that we can vote on the real thing and have prefix
implemented the right way before the end of the year.

>  6. (Any question that I've missed?)

Here are a few that I gathered from others (my comments are between
parentheses):

> How are dynamically linked set*id programs going to work?

> How are scripts using #!shebangs going to work?
> You write an ebuild, and you DEPEND upon >=foo-3, because the build
> process includes some foo code. The foo code is executed via
> scripts using #!/usr/bin/foo. Normally, this is fine.
> But on prefix, /usr/bin/foo might be a crappy, OS X mangled foo-2
> that's no good. So even though you've got the foo-3 dep met, it'll be
> met in /opt/Gentoo/blah, so your package will fail.

> How are ebuilds to be marked as supporting prefix or not?
(Here I'm guessing that changing the EAPI variable will do)

> Why is there only a single permitted installation path?
(I'm under the impression this is a limitation of the windows
installer but not of prefix itself. So patching the installer would
fix that)

> What exactly is expected from a prefix-compliant package manager to
> support full prefix installs, as opposed to just supporting installs
> to / with prefix-aware ebuilds?
(The PMS patch should answer that)

Denis.

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Jeremy Olexa]

Some questions answered. snipped the rest.

Denis Dupeyron wrote:
> 2009/10/18 Tomáš Chvátal <scarabeus@gentoo.org>:
>> Why on earth portage simply does not detect the prefix enviroment is being run
>> and then INTERNALY switch D->ED and other variables.
> 
> If that means we can get away without touching ebuilds, apart from
> changing their EAPI variable, then that's absolutely what we have to
> do. I'd like things to be done the right way though (see below).

When you change econf to do --prefix=${EPREFIX}/usr then you cannot 
simply s/D/ED/ for everything. I hope this makes sense when you think 
about it. ;)

   src_install() {
     emake DESTDIR="${D}" install || die
     mv "${ED}"/usr/bin/{,exuberant-}ctags || die
   }

But then again, some ebuilds need no changing once you fix econf to do 
the work, which is nice.

> 
> On Fri, Nov 13, 2009 at 4:43 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
>> However, there is need for additional discussion. From the council
>> meeting log I could extract the following open questions:
> 
> It would be preferable for the discussion to happen on this list
> before the meeting or we'll end up postponing again due to having more
> questions coming up at that time.

We are willing to talk, but it always seems like the Council is "not 
prepared" no matter what we do. Hope everyone involved can change that.

> 
>>  2. Should the Prefix team be allowed to do the necessary changes to
>>     ebuilds themselves, or should it be done by the respective
>>     maintainers?
> 
> I think here it's obvious that anybody who is an ebuild dev and sees
> anything to fix (prefix or else) is encouraged to go ahead and do it,
> as we've always done. The recommendation is and will always be to talk
> to the current maintainers out of politeness and to be extra careful
> (i.e. usually letting the maintainers do it) in case of
> system/tricky/exotic package. We don't give full cvs access to the
> whole tree to all ebuild devs for nothing.

It is quite obvious that we are not trying to make trouble. Talk is 
cheap, so we prefer that. But, we see no need to ask permission to add 
~prefix keywords, same as other arch teams.

Currently, 'repoman -d full' will fail in some packages. We are fixing this.

> Also I think it would be nice if somebody took care of a portage
> patch, since I hear it is rather simple. Fabian again? Or Zac? Any
> other volunteers?
> 
> I would prefer to have all the pieces in places before the next
> meeting so that we can vote on the real thing and have prefix
> implemented the right way before the end of the year.

portage devs and prefix devs have agreed that it is rather 'easy' to 
merge the prefix-portage branch. Just waiting.. ;) We have access to 
check into the portage repo, so this should not hold anything up 
regarding any decisions.

> 
>>  6. (Any question that I've missed?)

>> How are scripts using #!shebangs going to work?
>> You write an ebuild, and you DEPEND upon >=foo-3, because the build
>> process includes some foo code. The foo code is executed via
>> scripts using #!/usr/bin/foo. Normally, this is fine.
>> But on prefix, /usr/bin/foo might be a crappy, OS X mangled foo-2
>> that's no good. So even though you've got the foo-3 dep met, it'll be
>> met in /opt/Gentoo/blah, so your package will fail.

The prefix-portage branch has a nice feature that fixes shebangs 
automatically to be ${EPREFIX}/foo instead of /foo. It has even caught 
some Gentoo Linux bugs.

> 
>> How are ebuilds to be marked as supporting prefix or not?
> (Here I'm guessing that changing the EAPI variable will do)

Gentoo Prefix has keywords. So if EAPI 3 has ED/EROOT support but the 
ebuild doesn't use them then the ebuild does not need an EAPI bump. In 
this case, please rephrase your question to be "How are ebuilds to be 
marked as working on a prefix arch or not?" and then it is clear that it 
is the same as Gentoo Linux.

> 
>> Why is there only a single permitted installation path?
> (I'm under the impression this is a limitation of the windows
> installer but not of prefix itself. So patching the installer would
> fix that)

My installation path on my 6-8 prefix arches is in my NFS home. If you 
are referring to the Windows special installation package, well..that is 
just a "stage4" installer with binary packages. The windows installer is 
no where near the heart of Gentoo Prefix, instead it is a product of 
Gentoo Prefix and a convenience factor offered by another Prefix dev. It 
showcases the possibilities quite well, IMO.

You can set EPREFIX to anything. One of our users even set it to "/" - 
which we don't endorse but it is possible. :)

> 
>> What exactly is expected from a prefix-compliant package manager to
>> support full prefix installs, as opposed to just supporting installs
>> to / with prefix-aware ebuilds?
> (The PMS patch should answer that)
> 
> Denis.
> 

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 13-11-2009 12:43:25 +0100, Ulrich Mueller wrote:
> In its November meeting [1], the council has unanimously expressed
> support for this proposal [2].
> 
> However, there is need for additional discussion. From the council
> meeting log I could extract the following open questions:
> 
>   1. What are the implications for non-prefix devs and users? 

for devs: take note of EPREFIX, ED and EROOT, be aware what they mean
and eventually, use them in the right way.  This is not hard, since a
simple script[3] based on some heuristics can do the work in 98% of the
cases right.
for users: not an single bit, unless they look inside ebuilds

>   2. Should the Prefix team be allowed to do the necessary changes to
>      ebuilds themselves, or should it be done by the respective
>      maintainers?

The Prefix team is the equivalent of an arch team, only handling a big
bunch of arches [4].  Hence, keywording should be out of the question,
and just allowed.  Adding simple patches (to normal non-critical
ebuilds) used to be allowed AFAIR, so I see that as ok too.  In other
words, like darkside mentioned, I see our activities like those of an
arch team.

>   3. Are there any backwards compatibility or upgrade path issues for
>      eclasses that must still accept EAPI 0 (where the new ED, EROOT,
>      and EPREFIX variables are not defined)?

The backward and forward compatability is handled with conditional
code like:
  use prefix || local ED=${D}
or
  [[ -z ${ED} ]] && local ED=${D}

Since Zac checked in yesterday blacklisting of EPREFIX, EROOT and ED in
the main Portage branch, these variables hopefully soon become protected
in the sense that an externally set ED does not result in weird
behaviour of a Prefix aware eclass or ebuild when using the latter
approach, which is true forward compatible.
For your information, this was exactly what I asked for from the council.

>   4. EAPI numbering: Would this simply be added as an additional
>      feature to EAPI 3? Or should we have an intermediate EAPI slot,
>      e.g. 2.1 or 3 (and current EAPI 3 renamed to 4 in the latter
>      case)?

I don't care about this, but a fast EAPI is necessary to 1) rely on ED not
coming from outside, and 2) since we're defining ED, EROOT and EPREFIX
anyway, initialise ED and EROOT such that we don't need any conditional
code in ebuilds at all (and can just use ED where necessary).

>   5. Who is going to write the exact specification (PMS patch) for
>      this EAPI feature?

We agreed that I/Prefix team will give this a shot.

>   6. (Any question that I've missed?)

I guess most additional questions go beyond, and actually address the
real Prefix work, and its implications to Gentoo and many ebuilds, since
we have to touch around 2300 ebuilds, most with trivial changes, some
with heavy changes (think of the 32 patches that one needs to compile
Python...)

> Let's start the discussion now, in order to work out these details
> before the next council meeting (December 7th).
> 
> Ulrich
> 
> [1] <http://www.gentoo.org/proj/en/council/meeting-logs/20091109.txt>
>     (topic was discussed from 21:32 to 22:11 in the log's timezone)
> [2] <http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml>
> 

[3] http://overlays.gentoo.org/proj/alt/browser/trunk/prefix-overlay/scripts/eapify
[4] http://stats.prefix.freens.org/keywords-packages.png

-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 19-11-2009 19:42:11 -0600, Jeremy Olexa wrote:
> Some questions answered. snipped the rest.

readded questions where necessary

> Denis Dupeyron wrote:
> > 2009/10/18 Tomáš Chvátal <scarabeus@gentoo.org>:
> >> Why on earth portage simply does not detect the prefix enviroment is being run
> >> and then INTERNALY switch D->ED and other variables.
> > 
> > If that means we can get away without touching ebuilds, apart from
> > changing their EAPI variable, then that's absolutely what we have to
> > do. I'd like things to be done the right way though (see below).
> 
> When you change econf to do --prefix=${EPREFIX}/usr then you cannot 
> simply s/D/ED/ for everything. I hope this makes sense when you think 
> about it. ;)
> 
>    src_install() {
>      emake DESTDIR="${D}" install || die
>      mv "${ED}"/usr/bin/{,exuberant-}ctags || die
>    }
> 
> But then again, some ebuilds need no changing once you fix econf to do 
> the work, which is nice.
> 
> > 
> > On Fri, Nov 13, 2009 at 4:43 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
> >> However, there is need for additional discussion. From the council
> >> meeting log I could extract the following open questions:
> > 
> > It would be preferable for the discussion to happen on this list
> > before the meeting or we'll end up postponing again due to having more
> > questions coming up at that time.
> 
> We are willing to talk, but it always seems like the Council is "not 
> prepared" no matter what we do. Hope everyone involved can change that.
> 
> > 
> >>  2. Should the Prefix team be allowed to do the necessary changes to
> >>     ebuilds themselves, or should it be done by the respective
> >>     maintainers?
> > 
> > I think here it's obvious that anybody who is an ebuild dev and sees
> > anything to fix (prefix or else) is encouraged to go ahead and do it,
> > as we've always done. The recommendation is and will always be to talk
> > to the current maintainers out of politeness and to be extra careful
> > (i.e. usually letting the maintainers do it) in case of
> > system/tricky/exotic package. We don't give full cvs access to the
> > whole tree to all ebuild devs for nothing.
> 
> It is quite obvious that we are not trying to make trouble. Talk is 
> cheap, so we prefer that. But, we see no need to ask permission to add 
> ~prefix keywords, same as other arch teams.
> 
> Currently, 'repoman -d full' will fail in some packages. We are fixing this.

> >  4. EAPI numbering: Would this simply be added as an additional
> >     feature to EAPI 3? Or should we have an intermediate EAPI slot,
> >     e.g. 2.1 or 3 (and current EAPI 3 renamed to 4 in the latter
> >     case)?
> 
> Here I'd add to the choices: why not release an intermediate EAPI with
> the prefix stuff and whatever that has already been done for EAPI3?
> The exact name of a potential intermediate EAPI is a non-problem.
> However I would prefer if it were a number like 2.1 or 2.5 or even 3,
> because although we currently treat the EAPI variable as a simple
> string we may change our mind later and find it handy someday to use
> operators on them such as >=3D2.1.

While I think that this is a totally different discussion, unrelated to
Prefix (well, on a side-note, we once had things like EAPI="prefix 1",
which denoted both the features from "prefix" and "1"), it is good to
isolate things such that it is obvious what an EAPI stands for.

> >  5. Who is going to write the exact specification (PMS patch) for
> >     this EAPI feature?
> 
> I thought I asked Fabian to work on that at the end of the meeting. In
> case I didn't then consider this as me officially asking him if he can
> take care of it. Fabian is this OK with you?

Yes, I agreed coming up with some patch.  I admit I haven't yet even
looked into it.

> Also I think it would be nice if somebody took care of a portage

Technically, Portage trunk already contains the most important bits that
allow us to continue since yesterday.  The rest is waiting for a formal
approval of the council, and then it will most probably be me and Zac
fighting to get the prefix branch merged into trunk.

> > patch, since I hear it is rather simple. Fabian again? Or Zac? Any
> > other volunteers?
> > 
> > I would prefer to have all the pieces in places before the next
> > meeting so that we can vote on the real thing and have prefix
> > implemented the right way before the end of the year.
> 
> portage devs and prefix devs have agreed that it is rather 'easy' to 
> merge the prefix-portage branch. Just waiting.. ;) We have access to 
> check into the portage repo, so this should not hold anything up 
> regarding any decisions.

just to repeat: the implementation already exists in the "prefix" branch
of the portage repository.

> >>  6. (Any question that I've missed?)
> 
> >> How are scripts using #!shebangs going to work?
> >> You write an ebuild, and you DEPEND upon >=foo-3, because the build
> >> process includes some foo code. The foo code is executed via
> >> scripts using #!/usr/bin/foo. Normally, this is fine.
> >> But on prefix, /usr/bin/foo might be a crappy, OS X mangled foo-2
> >> that's no good. So even though you've got the foo-3 dep met, it'll be
> >> met in /opt/Gentoo/blah, so your package will fail.
> 
> The prefix-portage branch has a nice feature that fixes shebangs 
> automatically to be ${EPREFIX}/foo instead of /foo. It has even caught 
> some Gentoo Linux bugs.

Next to that, it is part of the Prefix team's job to make sure that
whatever is installed, does not reference the host system when this is
not absolutely necessary.  All of the questions you raise here, are
part of the job to get an ebuild "ported" from gx86 to Prefix.  In fact,
the "crappy, on OS X mangled foo-2" was the main drive to get the
project going.

> >> How are ebuilds to be marked as supporting prefix or not?
> > (Here I'm guessing that changing the EAPI variable will do)
> 
> Gentoo Prefix has keywords. So if EAPI 3 has ED/EROOT support but the 
> ebuild doesn't use them then the ebuild does not need an EAPI bump. In 
> this case, please rephrase your question to be "How are ebuilds to be 
> marked as working on a prefix arch or not?" and then it is clear that it 
> is the same as Gentoo Linux.
> 
> > 
> >> Why is there only a single permitted installation path?
> > (I'm under the impression this is a limitation of the windows
> > installer but not of prefix itself. So patching the installer would
> > fix that)
> 
> My installation path on my 6-8 prefix arches is in my NFS home. If you 
> are referring to the Windows special installation package, well..that is 
> just a "stage4" installer with binary packages. The windows installer is 
> no where near the heart of Gentoo Prefix, instead it is a product of 
> Gentoo Prefix and a convenience factor offered by another Prefix dev. It 
> showcases the possibilities quite well, IMO.
> 
> You can set EPREFIX to anything. One of our users even set it to "/" - 
> which we don't endorse but it is possible. :)
> 
> > 
> >> What exactly is expected from a prefix-compliant package manager to
> >> support full prefix installs, as opposed to just supporting installs
> >> to / with prefix-aware ebuilds?
> > (The PMS patch should answer that)

Agreed


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Denis Dupeyron]

Things seem to be progressing nicely on this front. We have answers to
the questions people had and they look satisfactory to me.

One thing that I think would be valuable is a document that explains
the average dev how to make his/her ebuilds prefix compliant with
links to more details when necessary. I understand that there's the
trivial situations and the less trivial ones. In the latter case it
would be nice to explain why the case isn't trivial and how to fix it.
Using python as an example could be one way to do it. I'm thinking of
something practical that could possibly be patched into devmanual. If
such a document already exists then please just point us to it.

On Fri, Nov 20, 2009 at 2:03 AM, Fabian Groffen <grobian@gentoo.org> wrote:
>> I thought I asked Fabian to work on that at the end of the meeting. In
>> case I didn't then consider this as me officially asking him if he can
>> take care of it. Fabian is this OK with you?
>
> Yes, I agreed coming up with some patch.  I admit I haven't yet even
> looked into it.

Great, thanks. If you can have it ready some time before the meeting
so that all devs can get a chance to review it before the council
members vote on it that will be even better. If you need help don't
hesitate to contact me. I'll try and look for the right people to help
you depending on what you need.

> Technically, Portage trunk already contains the most important bits that
> allow us to continue since yesterday.  The rest is waiting for a formal
> approval of the council, and then it will most probably be me and Zac
> fighting to get the prefix branch merged into trunk.

Good.

> Next to that, it is part of the Prefix team's job to make sure that
> whatever is installed, does not reference the host system when this is
> not absolutely necessary.

Could you give some examples of when it is absolutely necessary?

Denis.

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Denis Dupeyron]

It looks like this question is still unanswered:

On Thu, Nov 19, 2009 at 5:26 PM, Denis Dupeyron <calchan@gentoo.org> wrote:
>> How are dynamically linked set*id programs going to work?

Denis.

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 25-11-2009 16:43:32 -0700, Denis Dupeyron wrote:
> Things seem to be progressing nicely on this front. We have answers to
> the questions people had and they look satisfactory to me.
> 
> One thing that I think would be valuable is a document that explains
> the average dev how to make his/her ebuilds prefix compliant with
> links to more details when necessary. I understand that there's the
> trivial situations and the less trivial ones. In the latter case it
> would be nice to explain why the case isn't trivial and how to fix it.
> Using python as an example could be one way to do it. I'm thinking of
> something practical that could possibly be patched into devmanual. If
> such a document already exists then please just point us to it.

I'm working on a PMS patch at this very moment.  I hope to finish a
first version today.

The Python ebuild is a very bad example, IMO.  Not that the style there
is horrendous or something, but more that a couple of things are mixed
up in there, which are not directly a Prefix issue, but a derivative of
that -- support for arches which cannot be supported without Prefix.

We can explain some cases though.  The *blas-* family comes to mind as
we recently had to alter them for Prefix.

> > Next to that, it is part of the Prefix team's job to make sure that
> > whatever is installed, does not reference the host system when this is
> > not absolutely necessary.
> 
> Could you give some examples of when it is absolutely necessary?

Simple example is perl.  If you install a script, for instance ekeyword,
then it is important that this script doesn't say '#!/usr/bin/perl' in
its shebang.  "/usr/bin/perl" may simply not exist, but more importantly
it is not the perl that Portage has installed and also installed all
required dependencies for.  Hence, ekeyword should be installed such
that it references the perl from the offset installation, e.g.
"/home/joe/gentoo/usr/bin/perl".

"/bin/sh" is another nice one.  Technically this should work fine, as
almost every system has this shell available, however in practice it
won't work that well, because many people assume that "/bin/sh" is a
POSIX shell, which is not true on every UNIX that exists, as far as I
know.  People may also rely on things from the Gentoo environment in
such scripts, which of course are not present in the host provided
"/bin/sh", hence also here we need to call "/home/joe/gentoo/bin/sh"
(for the chosen offset-prefix "/home/joe/gentoo").

Outside scripts, also libraries have this same principle.  An
application linking against libexpat should do so against the Prefix
installed version of expat.  If the host system provides
"/usr/lib/libexpat.so" and the upstream configure script is hardwired to
look in "/usr/local/lib /usr/lib64 /usr/lib32 /usr/lib /usr/sfw/lib /sw"
it will never find libexpat from "/home/joe/gentoo/usr/lib".  In such
case either an --with-expat="${EPREFIX}"/usr option needs to be passed
to configure (if it exists), or if nothing possible configure be hacked
not to use this hardwired list of paths, but instead fall back on the
default, which for a Prefix toolchain includes the offset library search
paths before the host system's.  In any case, it must compile and link
against the Prefix installed libexpat, since the one from the host
system usually is very archaic and not under Portage's control.
 

-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 25-11-2009 17:01:19 -0700, Denis Dupeyron wrote:
> It looks like this question is still unanswered:
> 
> On Thu, Nov 19, 2009 at 5:26 PM, Denis Dupeyron <calchan@gentoo.org> wrote:
> >> How are dynamically linked set*id programs going to work?

Depends on how the host OS/libc handles this :)

If you're root, you can install a program that any user can call and
run as if that user were root.  If you're a normal user, you just
install a program that any user can call and run as if that user were
you.

If you refer to GNU glibc ignoring or greatly limiting variables like
LD_LIBRARY_PATH, then there is no problem in that for Prefix, since
these variables are considered harmful anyway.  We don't use them.
Prefix uses RUNPATH (RPATH) in ELF objects to have the loader find the
correct libraries at runtime without any tricks.  This is for instance
handy when someone calls a binary from a Prefix without having properly
initialised the Prefix environment.  While this is not what we advise
people to do, it can work fine for applications like subversion.


-- 
Fabian Groffen
Gentoo on a different level

[gentoo-dev] Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Duncan]

Fabian Groffen posted on Thu, 26 Nov 2009 09:53:04 +0100 as excerpted:

>> > Next to that, it is part of the Prefix team's job to make sure that
>> > whatever is installed, does not reference the host system when this
>> > is not absolutely necessary.
>> 
>> Could you give some examples of when it is absolutely necessary?
> 
> Simple example is perl.  If you install a script, for instance ekeyword,
> then it is important that this script doesn't say '#!/usr/bin/perl' in
> its shebang.  "/usr/bin/perl" may simply not exist, but more importantly
> it is not the perl that Portage has installed and also installed all
> required dependencies for.  Hence, ekeyword should be installed such
> that it references the perl from the offset installation, e.g.
> "/home/joe/gentoo/usr/bin/perl".
> 
> "/bin/sh" is another nice one.

At least here, that it would ordinarily be best to reference the prefix 
system was taken for granted, so when it's "absolutely necessary" to 
reference the host system is the interesting case, and how I parsed the 
request.  You provided examples of just the opposite, the case I (and 
evidently Denis, if I parsed the request correctly) assumed to be normal, 
where referencing the prefix is strongly desirable or "absolutely 
necessary".

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 26-11-2009 10:01:24 +0000, Duncan wrote:
> > required dependencies for.  Hence, ekeyword should be installed such
> > that it references the perl from the offset installation, e.g.
> > "/home/joe/gentoo/usr/bin/perl".
> > 
> > "/bin/sh" is another nice one.
> 
> At least here, that it would ordinarily be best to reference the prefix 
> system was taken for granted, so when it's "absolutely necessary" to 
> reference the host system is the interesting case, and how I parsed the 
> request.  You provided examples of just the opposite, the case I (and 
> evidently Denis, if I parsed the request correctly) assumed to be normal, 
> where referencing the prefix is strongly desirable or "absolutely 
> necessary".

I see, thanks for bringing that up.  To clarify (I hope this is):

Gentoo Prefix tries to be as much self-sufficient as possible, and hence
applications *must* not reference the host system, unless absolutely
necessary, such as for e.g. /lib/libc.so.

If Gentoo Prefix would not do this, Portage's dependencies would become
useless, and many uncontrollable and unpredictable errors may arise both
at compile time as well as at run time.


-- 
Fabian Groffen
Gentoo on a different level

[gentoo-dev] Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Duncan]

Fabian Groffen posted on Thu, 26 Nov 2009 11:10:09 +0100 as excerpted:

> Gentoo Prefix tries to be as much self-sufficient as possible, and hence
> applications *must* not reference the host system, unless absolutely
> necessary, such as for e.g. /lib/libc.so.

Thanks.  Host libc /does/ make sense as "absolutely necessary.

Are there any less obvious ones, say of the type that might reach out and 
grab an unsuspecting dev trying to make his ebuilds prefix compliant?  It 
seems to me that enumerating all (or all non-corner) cases where 
referencing the host is desired/mandatory, with a blanket rule saying 
reference prefix unless it's a known exception, should be by /far/ the 
easiest alternative, here.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 26-11-2009 10:37:10 +0000, Duncan wrote:
> Fabian Groffen posted on Thu, 26 Nov 2009 11:10:09 +0100 as excerpted:
> 
> > Gentoo Prefix tries to be as much self-sufficient as possible, and hence
> > applications *must* not reference the host system, unless absolutely
> > necessary, such as for e.g. /lib/libc.so.
> 
> Thanks.  Host libc /does/ make sense as "absolutely necessary.
> 
> Are there any less obvious ones, say of the type that might reach out and 

Some that you may find are:
/lib/libm.so
/lib/libsocket.so
/lib/libpthread.so
/lib/libnsl.so

On a side note, we have a question about this in our
prefix-ebuild-quiz[1] (question 5 from the second section).

> grab an unsuspecting dev trying to make his ebuilds prefix compliant?  It 
> seems to me that enumerating all (or all non-corner) cases where 
> referencing the host is desired/mandatory, with a blanket rule saying 
> reference prefix unless it's a known exception, should be by /far/ the 
> easiest alternative, here.

I think there's unfortunately no simple way to tell what should be in
and what unfortunately has to be out.  It depends a lot on the host
system.  I feel -- but I can't back this up with hard evidence -- that
it are usually the libs that are not in *DEPEND that can only be
available in the host system.  Basically because they usually are part
of the libc, which we assume to be installed.


[1] http://dev.gentoo.org/~grobian/prefix-quiz

-- 
Fabian Groffen
Gentoo on a different level

[gentoo-dev] Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Duncan]

Fabian Groffen posted on Thu, 26 Nov 2009 11:51:06 +0100 as excerpted:

>> Are there any less obvious ones
> 
> Some that you may find are:
> /lib/libm.so
> /lib/libsocket.so
> /lib/libpthread.so
> /lib/libnsl.so
> 
> On a side note, we have a question about this in our
> prefix-ebuild-quiz[1] (question 5 from the second section).

> I think there's unfortunately no simple way to tell what should be in
> and what unfortunately has to be out.  It depends a lot on the host
> system.  I feel -- but I can't back this up with hard evidence -- that
> it are usually the libs that are not in *DEPEND that can only be
> available in the host system.  Basically because they usually are part
> of the libc, which we assume to be installed.
> 
> 
> [1] http://dev.gentoo.org/~grobian/prefix-quiz

Thanks again.  That quiz was particularly enlightening (to this gentoo, 
not prefix, user) on the type of stuff you think about, as a part of what 
prefix /does/.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

On 25-11-2009 16:43:32 -0700, Denis Dupeyron wrote:
> > Yes, I agreed coming up with some patch.  I admit I haven't yet even
> > looked into it.
> 
> Great, thanks. If you can have it ready some time before the meeting
> so that all devs can get a chance to review it before the council
> members vote on it that will be even better. If you need help don't
> hesitate to contact me. I'll try and look for the right people to help
> you depending on what you need.

Here is the patch.  I tried to cover all places.


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: prefix-r0.patch --]
[-- Type: text/plain, Size: 14760 bytes --]

diff --git a/eapi-differences.tex b/eapi-differences.tex
index 7e792b8..2fdf763 100644
--- a/eapi-differences.tex
+++ b/eapi-differences.tex
@@ -176,7 +176,8 @@ Use dependencies & \compactfeatureref{use-deps} & No & No &
     \IFANYKDEBUILDELSE{\IFKDEBUILDCOLOUR{None} &}{}
     \parbox[t]{1in}{\t{pkg\_nofetch}, \t{src\_unpack}, \t{src\_prepare}, \t{src\_configure}, \t{src\_compile}, \t{src\_test}} &
     \parbox[t]{1in}{\t{pkg\_nofetch}, \t{src\_unpack}, \t{src\_prepare}, \t{src\_configure},
-        \t{src\_compile}, \t{src\_install}, \t{src\_test}} \\
+        \t{src\_compile}, \t{src\_install}, \t{src\_test},
+        \t{EPREFIX}, \t{ED}, \t{EROOT}} \\
 
 \t{AA} & \compactfeatureref{aa} & Yes & Yes &
     \IFANYKDEBUILDELSE{\IFKDEBUILDCOLOUR{Yes} &}{} Yes & No \\
@@ -335,6 +336,8 @@ EAPI 3 is EAPI 2 with the following changes:
 \item \t{RDEPEND=DEPEND} no longer done, \featureref{rdepend-depend}.
 \item Utilities now die on failure, \featureref{die-on-failure}, unless called under \t{nonfatal},
     \featureref{nonfatal}
+\item offset-prefix support by definition of \t{EPREFIX}, \t{ED} and
+\t{EROOT}, \featureref{offset-prefix-vars}
 \end{compactitem}
 
 % vim: set filetype=tex fileencoding=utf8 et tw=100 spell spelllang=en :
diff --git a/ebuild-env-vars.tex b/ebuild-env-vars.tex
index e093218..3f9d25c 100644
--- a/ebuild-env-vars.tex
+++ b/ebuild-env-vars.tex
@@ -13,7 +13,7 @@ variable.
 
 \begin{landscape}
 \begin{longtable}{l p{0.15\textwidth} l p{0.5\textwidth}}
-\caption{Defined variables}\\
+\caption{Defined variables\label{tab:defined_vars}}\\
 \toprule
 \multicolumn{1}{c}{\b{Variable}} &
 \multicolumn{1}{c}{\b{Legal in}} &
@@ -104,6 +104,11 @@ variable.
    \t{ROOT}\@. Also of note is that in a cross-compiling environment, binaries inside of \t{ROOT}
    will not be executable on the build machine, so ebuilds must not call them. \t{ROOT} must be
    non-empty and end in a trailing slash. \\
+\t{EROOT} &
+    \t{pkg\_*} &
+	No &
+	Like \t{ROOT}, but with \t{EPREFIX} appended.  This is a convenience
+	variable.  See also the \t{EPREFIX} variable. \\
 \t{T} &
     All &
     Partially\footnote{Consistent and preserved across a single connected sequence of install or
@@ -120,6 +125,17 @@ variable.
     Ditto &
     The full path to an appropriate temporary directory for use by any programs invoked by the
     ebuild that may read or modify the home directory. \\
+\t{EPREFIX} &
+    All &
+	Yes &
+	The normalised offset-prefix path of an offset installation.  When
+	\t{EPREFIX} is not set in the calling environment, \t{EPREFIX}
+	defaults to the built in offset-prefix that was set during
+	installation of the package manager.  When a different \t{EPREFIX}
+	value than the built in value is set in the environment, a
+	cross-prefix build is performed where using the existing utilities,
+	a package is build for the given \t{EPREFIX}, akin to \t{ROOT}.  See
+	also~\ref{sec:offset-vars}. \\
 \t{D} &
     \t{src\_install} &
     No &
@@ -130,6 +146,11 @@ variable.
     Yes &
     Contains the full path to the image that is about to be or has just been merged. Must be
     non-empty and end in a trailing slash. \\
+\t{ED} &
+    \t{src\_install} &
+	See \t{D} &
+	Like \t{D}, but with \t{EPREFIX} appended.  This is a convenience
+	variable.  See also the \t{EPREFIX} variable. \\
 \t{IMAGE}\footnote{Deprecated in favour of \t{D}.} &
     \t{pkg\_preinst}, \t{pkg\_postinst} &
     Yes &
@@ -216,6 +237,22 @@ variable.
 }
 \end{centertable}
 
+\begin{centertable}{EAPIs supporting offset-prefix env variables} \label{tab:offset-env-vars-table}
+\begin{tabular}{ l l l l }
+	\toprule
+	\multicolumn{1}{c}{\textbf{EAPI}} &
+	\multicolumn{1}{c}{\textbf{\t{EPREFIX}?}} &
+	\multicolumn{1}{c}{\textbf{\t{EROOT}?}} &
+	\multicolumn{1}{c}{\textbf{\t{ED}?}} \\
+	\midrule
+\t{0} & No & No & No \\
+\t{1} & No & No & No \\
+\t{2} & No & No & No \\
+\t{3} & Yes & Yes & Yes \\
+\bottomrule
+\end{tabular}
+\end{centertable}
+
 Except where otherwise noted, all variables set in the active profiles' \t{make.defaults} files must
 be exported to the ebuild environment. \t{CHOST}, \t{CBUILD} and \t{CTARGET}, if not set by
 profiles, must contain either an appropriate machine tuple (the definition of appropriate is beyond
@@ -309,6 +346,18 @@ installing \t{foo-2:2} to replace \t{foo-2:1} and \t{foo-3:2}.
 In EAPIs listed in table~\ref{tab:env-vars-table} as supporting it, the \t{REPLACED\_BY} variable
 shall be defined in \t{pkg\_prerm} and \t{pkg\_postrm}. It shall contain at most one value.
 
+\subsection{Offset-prefix variables \t{EPREFIX}, \t{EROOT} and \t{ED}}
+\label{sec:offset-vars}
+
+\featurelabel{offset-prefix-vars} In EAPI 3, three variables related to
+offset-prefix installations were added.  Two of these, \t{EROOT} and
+\t{ED}, are convenience variables using the variable \t{EPREFIX}.  In
+EAPIs prior to 3, the installation offset is hardwired to \t{/usr}.
+Using the variable \t{EPREFIX}, in EAPI 3 this installation offset can
+be set as \t{\$\{EPREFIX\}/usr}.  This way, \t{EPREFIX} set to the empty
+string makes the behaviour identical to EAPIs prior to 3, but adds two
+extra variables.
+
 % vim: set filetype=tex fileencoding=utf8 et tw=100 spell spelllang=en :
 
 %%% Local Variables:
diff --git a/pkg-mgr-commands.tex b/pkg-mgr-commands.tex
index 4b88b20..de2f325 100644
--- a/pkg-mgr-commands.tex
+++ b/pkg-mgr-commands.tex
@@ -169,12 +169,12 @@ has returned.
 
     \featurelabel{econf-options}
     \begin{itemize}
-    \item -{}-prefix must default to \t{/usr} unless overridden by \t{econf}'s caller.
-    \item -{}-mandir must be \t{/usr/share/man}
-    \item -{}-infodir must be \t{/usr/share/info}
-    \item -{}-datadir must be \t{/usr/share}
-    \item -{}-sysconfdir must be \t{/etc}
-    \item -{}-localstatedir must be \t{/var/lib}
+    \item -{}-prefix must default to \t{\$EPREFIX/usr} unless overridden by \t{econf}'s caller.
+    \item -{}-mandir must be \t{\$EPREFIX/usr/share/man}
+    \item -{}-infodir must be \t{\$EPREFIX/usr/share/info}
+    \item -{}-datadir must be \t{\$EPREFIX/usr/share}
+    \item -{}-sysconfdir must be \t{\$EPREFIX/etc}
+    \item -{}-localstatedir must be \t{\$EPREFIX/var/lib}
     \item -{}-host must be the value of the \t{CHOST} environment variable.
     \item -{}-libdir must be set according to Algorithm~\ref{alg:econf-libdir}.
     \item -{}-disable-dependency-tracking, if the EAPI is listed in
@@ -211,6 +211,13 @@ has returned.
     }
     \end{centertable}
 
+	Note that the \t{\$EPREFIX} component represents the same
+	offset-prefix as described in Table~\ref{tab:defined_vars}.  It
+	facilitates offset-prefix installations for which support was added
+	in EAPI 3.  When no offset-prefix installation is in effect,
+	\t{EPREFIX} becomes the empty string, making the behaviour of econf
+	equal to EAPIs prior to EAPI 3.
+
     \t{econf} must be implemented internally---that is, as a bash function and not an external
     script. Should any portion of it fail, it must abort the build using \t{die}, unless run using
     \t{nonfatal}, in which case it must return non-zero exit status.
@@ -218,7 +225,7 @@ has returned.
 \begin{algorithm}
 \caption{econf -{}-libdir logic} \label{alg:econf-libdir}
 \begin{algorithmic}[1]
-\STATE let prefix=/usr
+\STATE let prefix=\$EPREFIX/usr
 \IF{the caller specified -{}-prefix=\$p}
     \STATE let prefix=\$p
 \ENDIF
@@ -245,14 +252,19 @@ has returned.
     to \t{einstall} are passed verbatim to \t{emake}, as shown. Failure behaviour is EAPI dependent
     as per section~\ref{sec:failure-behaviour}.
 
+	The variable \t{ED} is defined as in Table~\ref{tab:defined_vars}
+	and depends on the use of an offset-prefix.  When such offset-prefix
+	is absent, \t{ED} is equivalent to \t{D}.  \t{ED} was introduced in
+	EAPI 3, hence in prior EAPIs, \t{D} should be used instead of \t{ED}
+	in the command given in Listing~\ref{lst:einstall}.
 \begin{listing}[H]
   \caption{einstall command}\label{lst:einstall}
   \begin{verbatim}
 emake \
-   prefix="${D}"/usr \
-   mandir="${D}"/usr/share/man \
-   infodir="${D}"/usr/share/info \
-   libdir="${D}"/usr/$(get_libdir) \
+   prefix="${ED}"/usr \
+   mandir="${ED}"/usr/share/man \
+   infodir="${ED}"/usr/share/info \
+   libdir="${ED}"/usr/$(get_libdir) \
    "$@" \
    install
   \end{verbatim}
@@ -263,7 +275,8 @@ emake \
 \subsubsection{Installation commands}
 These commands are used to install files into the staging area, in cases where the package's \t{make
 install} target cannot be used or does not install all needed files. Except where otherwise stated,
-all filenames created or modified are relative to the staging directory, given by \t{D}. These
+all filenames created or modified are relative to the staging directory,
+given by \t{ED} for EAPI 3, and \t{D} for EAPIs prior to EAPI 3.  These
 commands must all be external programs and not bash functions or aliases---that is, they must be
 callable from \t{xargs}. Ebuilds must not run any of these commands once the current phase function
 has returned.
@@ -272,6 +285,9 @@ has returned.
 \item[dobin] Installs the given files into \t{DESTTREE/bin}, where \t{DESTTREE} defaults to
     \t{/usr}. Gives the files mode \t{0755} and ownership \t{root:root}. Failure behaviour is EAPI
     dependent as per section~\ref{sec:failure-behaviour}.
+% todo: Portage does not behave like this, and Prefix relies on that
+%       root:root -> 0:0 (for systems where root name/group are different)
+%       PORTAGE_INST_UID:PORTAGE_INST_GID <- Prefix sets that to user ids
 
 \item[doconfd] Installs the given config files into \t{/etc/conf.d/}, by default with file mode
     \t{0644}. This can be overridden by setting \t{INSOPTIONS} with the \t{insopts} function.
@@ -317,7 +333,8 @@ that can be passed to \t{dohtml} are as follows:
     \item{\t{-f}} --- list of files that are able to be installed.
     \item{\t{-x}} --- list of directories that files will not be installed from (only used in
     conjunction with \t{-r}).
-    \item{\t{-p}} --- sets a document prefix for installed files.
+    \item{\t{-p}} --- sets a document prefix for installed files, not to
+	be confused with the global offset-prefix.
     \end{description}
 
     Failure behaviour is EAPI dependent as per section~\ref{sec:failure-behaviour}.
@@ -556,7 +573,9 @@ that can be passed to \t{dohtml} are as follows:
 }
 
 \subsubsection{Commands affecting install destinations}
-The following commands are used to set the various destination trees, all relative to \t{\$\{D\}},
+The following commands are used to set the various destination trees,
+all relative to \t{\$\{ED\}} in EAPI 3 and relative to \t{\$\{D\}} in
+prior EAPIs,
 used by the above installation commands. They must be shell functions or aliases, due to the need to
 set variables read by the above commands. Ebuilds must not run any of these commands once the
 current phase function has returned.
@@ -564,7 +583,8 @@ current phase function has returned.
 \begin{description}
 
 \item[into] Sets the value of \t{DESTTREE} for future invocations of the above utilities.  Creates
-the directory under \t{\$\{D\}}, using \t{install -d} with no additional options, if it does not
+the directory under \t{\$\{ED\}} for EAPI 3 or \t{\$\{D\}} for prior
+EAPIs, using \t{install -d} with no additional options, if it does not
 already exist. Failure behaviour is EAPI dependent as per section~\ref{sec:failure-behaviour}.
 
 \item[insinto] Sets the value of \t{INSDESTTREE} for future invocations of the above utilities. May
@@ -590,7 +610,7 @@ already exist. Failure behaviour is EAPI dependent as per section~\ref{sec:failu
 
 \featurelabel{controllable-compress} In EAPIs listed in table~\ref{tab:compression-table} as
 supporting controllable compression, the package manager may optionally compress a subset of the
-files under the \t{D} directory. To control which directories may or may not be compressed, the
+files under the \t{ED} for EAPI 3 and \t{D} for prior EAPIs directory. To control which directories may or may not be compressed, the
 package manager shall maintain two lists:
 
 \begin{compactitem}
@@ -601,7 +621,8 @@ package manager shall maintain two lists:
 
 The optional compression shall be carried out after \t{src\_install} has completed, and before the
 execution of any subsequent phase function. For each item in the inclusion list, pretend it has the
-value of the \t{D} variable prepended, then:
+value of the \t{ED} variable in EAPI 3 or the \t{D} variable in prior
+EAPIs prepended, then:
 
 \begin{compactitem}
 \item If it is a directory, act as if every file or directory immediately under this directory
@@ -611,7 +632,8 @@ value of the \t{D} variable prepended, then:
 \end{compactitem}
 
 Whether an item is to be excluded is determined as follows: For each item in the exclusion list,
-pretend it has the value of the \t{D} variable prepended, then:
+pretend it has the value of the \t{ED} variable in EAPI 3 or the \t{D}
+variable in prior EAPIs prepended, then:
 
 \begin{compactitem}
 \item If it is a directory, act as if every file or directory immediately under this directory
@@ -736,9 +758,11 @@ has returned.
 
 \begin{description}
 \item[dosed] Takes any number of arguments, which can be files or \t{sed} expressions. For each
-    argument, if it names, relative to \t{D} a file which exists, then \t{sed} is run with the
+    argument, if it names, relative to \t{ED} (EAPI 3) or \t{D} (EAPIs
+	prior to 3) a file which exists, then \t{sed} is run with the
     current expression on that file. Otherwise, the current expression is set to the text of the
-    argument. The initial value of the expression is \t{s:\$\{D\}::g}. In EAPIs listed in
+    argument. The initial value of the expression is \t{s:\$\{ED\}::g}
+	in EAPI 3 and \t{s:\$\{D\}::g} in prior EAPIs. In EAPIs listed in
     table~\ref{tab:banned-commands-table}, this command is banned as per
     section~\ref{sec:banned-commands}. Failure behaviour is EAPI dependent as per
     section~\ref{sec:failure-behaviour}.
@@ -771,6 +795,9 @@ has returned.
     \item LHA archives (\t{*.LHA, *.LHa, *.lha, *.lhz}). Ebuilds must ensure that the lha program is
     installed.
     \item ar archives (\t{*.a, *.deb}). Ebuilds must ensure that GNU binutils is installed.
+% todo: Portage's implementation is different and required for Prefix
+%       these files are unpacked with deb2targz if available, else it
+%       falls back to "ar", which cannot be required to be GNU binutils
     \item lzma-compressed files (\t{*.lzma}). Ebuilds must ensure that LZMA Utils is installed.
     \item lzma-compressed tar files (\t{*.tar.lzma}). Ebuilds must ensure that LZMA Utils and
         GNU tar are installed.

Re: [gentoo-dev] Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

[Fabian Groffen]

On 26-11-2009 12:36:47 +0000, Duncan wrote:
> > I think there's unfortunately no simple way to tell what should be in
> > and what unfortunately has to be out.  It depends a lot on the host
> > system.  I feel -- but I can't back this up with hard evidence -- that
> > it are usually the libs that are not in *DEPEND that can only be
> > available in the host system.  Basically because they usually are part
> > of the libc, which we assume to be installed.
> > 
> > 
> > [1] http://dev.gentoo.org/~grobian/prefix-quiz
> 
> Thanks again.  That quiz was particularly enlightening (to this gentoo, 
> not prefix, user) on the type of stuff you think about, as a part of what 
> prefix /does/.

The following bug is a classical example of something which just happens
to work by accident, but is broken.  It also illustrates what kind of
changes people can expect to have to be made to their ebuilds to get
them to work for Prefix:

  http://bugs.gentoo.org/show_bug.cgi?id=293901

Because the ebuild gives a direct pointer to a place in the filesystem,

  --with-apr=/usr/bin/apr-1-config

the ebuild needs a change such that it points to the corresponding place
in the offset-prefix installation, e.g. under "${EPREFIX}".

  --with-apr="${EPREFIX}"/usr/bin/apr-1-config

When this variable is empty, there's obviously no change to the
behaviour of the ebuild.


-- 
Fabian Groffen
Gentoo on a different level